You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Gerard Maas <ge...@gmail.com> on 2014/05/21 14:47:22 UTC
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Hi Tobias,
I was curious about this issue and tried to run your example on my local
Mesos. I was able to reproduce your issue using your current config:
[error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
1.0:4 failed 4 times (most recent failure: Exception failure:
java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2)
org.apache.spark.SparkException: Job aborted: Task 1.0:4 failed 4 times
(most recent failure: Exception failure: java.lang.ClassNotFoundException:
spark.SparkExamplesMinimal$$anonfun$2)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
Creating a simple jar from the job and providing it through the
configuration seems to solve it:
val conf = new SparkConf()
.setMaster("mesos://<my_ip>:5050/")
*
.setJars(Seq("/sparkexample/target/scala-2.10/sparkexample_2.10-0.1.jar"))*
.setAppName("SparkExamplesMinimal")
Resulting in:
14/05/21 12:03:45 INFO scheduler.DAGScheduler: Completed ResultTask(1, 1)
14/05/21 12:03:45 INFO scheduler.DAGScheduler: Stage 1 (count at
SparkExamplesMinimal.scala:50) finished in 1.120 s
14/05/21 12:03:45 INFO spark.SparkContext: Job finished: count at
SparkExamplesMinimal.scala:50, took 1.177091435 s
count: 1000000
Why the closure serialization does not work with Mesos is beyond my current
knowledge.
Would be great to hear from the experts (cross-posting to dev for that)
-kr, Gerard.
On Wed, May 21, 2014 at 11:51 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> I have set up a cluster with Mesos (backed by Zookeeper) with three
> master and three slave instances. I set up Spark (git HEAD) for use
> with Mesos according to this manual:
> http://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
>
> Using the spark-shell, I can connect to this cluster and do simple RDD
> operations, but the same code in a Scala class and executed via sbt
> run-main works only partially. (That is, count() works, count() after
> flatMap() does not.)
>
> Here is my code: https://gist.github.com/tgpfeiffer/7d20a4d59ee6e0088f91
> The file SparkExamplesScript.scala, when pasted into spark-shell,
> outputs the correct count() for the parallelized list comprehension,
> as well as for the flatMapped RDD.
>
> The file SparkExamplesMinimal.scala contains exactly the same code,
> and also the MASTER configuration and the Spark Executor are the same.
> However, while the count() for the parallelized list is displayed
> correctly, I receive the following error when asking for the count()
> of the flatMapped RDD:
>
> -----------------
>
> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting Stage 1
> (FlatMappedRDD[1] at flatMap at SparkExamplesMinimal.scala:34), which
> has no missing parents
> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting 8 missing
> tasks from Stage 1 (FlatMappedRDD[1] at flatMap at
> SparkExamplesMinimal.scala:34)
> 14/05/21 09:47:49 INFO scheduler.TaskSchedulerImpl: Adding task set
> 1.0 with 8 tasks
> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Starting task 1.0:0
> as TID 8 on executor 20140520-102159-2154735808-5050-1108-1: mesos9-1
> (PROCESS_LOCAL)
> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Serialized task 1.0:0
> as 1779147 bytes in 37 ms
> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Lost TID 8 (task 1.0:0)
> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Loss was due to
> java.lang.ClassNotFoundException
> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> at
> org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
> at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
> at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> -----------------
>
> Can anyone explain to me where this comes from or how I might further
> track the problem down?
>
> Thanks,
> Tobias
>
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Gerard Maas <ge...@gmail.com>.
Hi Andrew,
Thanks for the current doc.
I'd almost gotten to the point where I thought that my custom code needed
> to be included in the SPARK_EXECUTOR_URI but that can't possibly be
> correct. The Spark workers that are launched on Mesos slaves should start
> with the Spark core jars and then transparently get classes from custom
> code over the network, or at least that's who I thought it should work.
> For those who have been using Mesos in previous releases, you've never had
> to do that before have you?
Regarding the delivery of the custom job code to Mesos, we have been using
'ADD_JARS' (in the command line) or 'SparkConfig.setJars(Seq[String]) with
a fat jar packing all dependencies.
That works as well on the Spark 'standalone' cluster, but we deploy mostly
on Mesos, so I couldn't say about classloading difference between the two.
-greetz, Gerard.
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Andrew Ash <an...@andrewash.com>.
Here's the 1.0.0rc9 version of the docs:
https://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/running-on-mesos.html
I refreshed them with the goal of steering users more towards prebuilt
packages than relying on compiling from source plus improving overall
formatting and clarity, but not otherwise modifying the content. I don't
expect any changes for rc10.
It does seem like an issue though that classpath issues are preventing that
from running. Just to check, have you given the exact some jar a shot when
running against a standalone cluster? If it works in standalone, I think
that's good evidence that there's an issue with the Mesos classloaders in
master.
I'm running into a similar issue with classpaths failing on Mesos but
working in standalone, but I haven't coherently written up my observations
yet so haven't gotten that to this list.
I'd almost gotten to the point where I thought that my custom code needed
to be included in the SPARK_EXECUTOR_URI but that can't possibly be
correct. The Spark workers that are launched on Mesos slaves should start
with the Spark core jars and then transparently get classes from custom
code over the network, or at least that's who I thought it should work.
For those who have been using Mesos in previous releases, you've never had
to do that before have you?
On Wed, May 21, 2014 at 3:30 PM, Gerard Maas <ge...@gmail.com> wrote:
> Hi Tobias,
>
> On Wed, May 21, 2014 at 5:45 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> >first, thanks for your explanations regarding the jar files!
> No prob :-)
>
>
>> On Thu, May 22, 2014 at 12:32 AM, Gerard Maas <ge...@gmail.com>
>> wrote:
>> > I was discussing it with my fellow Sparkers here and I totally
>> overlooked
>> > the fact that you need the class files to de-serialize the closures (or
>> > whatever) on the workers, so you always need the jar file delivered to
>> the
>> > workers in order for it to work.
>>
>> So the closure as a function is serialized, sent across the wire,
>> deserialized there, and *still* you need the class files? (I am not
>> sure I understand what is actually sent over the network then. Does
>> that serialization only contain the values that I close over?)
>>
>
> I also had that mental lapse. Serialization refers to converting object
> (not class) state (current values) into a byte stream and de-serialization
> restores the bytes from the wire into an seemingly identical object at the
> receiving side (except for transient variables), for that, it requires the
> class definition of that object to know what it needs to instantiate, so
> yes, the compiled classes need to be given to the Spark driver and it will
> take care of dispatching them to the workers (much better than in the old
> RMI days ;-)
>
>
>> If I understand correctly what you are saying, then the documentation
>> at
>> https://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
>> (list item 8) needs to be extended quite a bit, right?
>>
>
> The mesos docs have been recently updated here:
> https://github.com/apache/spark/pull/756/files
> Don't know where the latest version from master is built/available.
>
> -kr, Gerard.
>
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Gerard Maas <ge...@gmail.com>.
Hi Tobias,
On Wed, May 21, 2014 at 5:45 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>first, thanks for your explanations regarding the jar files!
No prob :-)
> On Thu, May 22, 2014 at 12:32 AM, Gerard Maas <ge...@gmail.com>
> wrote:
> > I was discussing it with my fellow Sparkers here and I totally overlooked
> > the fact that you need the class files to de-serialize the closures (or
> > whatever) on the workers, so you always need the jar file delivered to
> the
> > workers in order for it to work.
>
> So the closure as a function is serialized, sent across the wire,
> deserialized there, and *still* you need the class files? (I am not
> sure I understand what is actually sent over the network then. Does
> that serialization only contain the values that I close over?)
>
I also had that mental lapse. Serialization refers to converting object
(not class) state (current values) into a byte stream and de-serialization
restores the bytes from the wire into an seemingly identical object at the
receiving side (except for transient variables), for that, it requires the
class definition of that object to know what it needs to instantiate, so
yes, the compiled classes need to be given to the Spark driver and it will
take care of dispatching them to the workers (much better than in the old
RMI days ;-)
> If I understand correctly what you are saying, then the documentation
> at https://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
> (list item 8) needs to be extended quite a bit, right?
>
The mesos docs have been recently updated here:
https://github.com/apache/spark/pull/756/files
Don't know where the latest version from master is built/available.
-kr, Gerard.
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi Gerard,
first, thanks for your explanations regarding the jar files!
On Thu, May 22, 2014 at 12:32 AM, Gerard Maas <ge...@gmail.com> wrote:
> I was discussing it with my fellow Sparkers here and I totally overlooked
> the fact that you need the class files to de-serialize the closures (or
> whatever) on the workers, so you always need the jar file delivered to the
> workers in order for it to work.
So the closure as a function is serialized, sent across the wire,
deserialized there, and *still* you need the class files? (I am not
sure I understand what is actually sent over the network then. Does
that serialization only contain the values that I close over?)
If I understand correctly what you are saying, then the documentation
at https://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
(list item 8) needs to be extended quite a bit, right?
Thanks
Tobias
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Gerard Maas <ge...@gmail.com>.
Hi Tobias,
Regarding my comment on closure serialization:
I was discussing it with my fellow Sparkers here and I totally overlooked
the fact that you need the class files to de-serialize the closures (or
whatever) on the workers, so you always need the jar file delivered to the
workers in order for it to work.
The SparkREPL works differently. It uses some dark magic to send the
working session to the workers.
-kr, Gerard.
On Wed, May 21, 2014 at 2:47 PM, Gerard Maas <ge...@gmail.com> wrote:
> Hi Tobias,
>
> I was curious about this issue and tried to run your example on my local
> Mesos. I was able to reproduce your issue using your current config:
>
> [error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
> 1.0:4 failed 4 times (most recent failure: Exception failure:
> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2)
> org.apache.spark.SparkException: Job aborted: Task 1.0:4 failed 4 times
> (most recent failure: Exception failure: java.lang.ClassNotFoundException:
> spark.SparkExamplesMinimal$$anonfun$2)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
>
> Creating a simple jar from the job and providing it through the
> configuration seems to solve it:
>
> val conf = new SparkConf()
> .setMaster("mesos://<my_ip>:5050/")
> *
> .setJars(Seq("/sparkexample/target/scala-2.10/sparkexample_2.10-0.1.jar"))*
> .setAppName("SparkExamplesMinimal")
>
> Resulting in:
> 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Completed ResultTask(1, 1)
> 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Stage 1 (count at
> SparkExamplesMinimal.scala:50) finished in 1.120 s
> 14/05/21 12:03:45 INFO spark.SparkContext: Job finished: count at
> SparkExamplesMinimal.scala:50, took 1.177091435 s
> count: 1000000
>
> Why the closure serialization does not work with Mesos is beyond my
> current knowledge.
> Would be great to hear from the experts (cross-posting to dev for that)
>
> -kr, Gerard.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, May 21, 2014 at 11:51 AM, Tobias Pfeiffer <tg...@preferred.jp>wrote:
>
>> Hi,
>>
>> I have set up a cluster with Mesos (backed by Zookeeper) with three
>> master and three slave instances. I set up Spark (git HEAD) for use
>> with Mesos according to this manual:
>> http://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
>>
>> Using the spark-shell, I can connect to this cluster and do simple RDD
>> operations, but the same code in a Scala class and executed via sbt
>> run-main works only partially. (That is, count() works, count() after
>> flatMap() does not.)
>>
>> Here is my code: https://gist.github.com/tgpfeiffer/7d20a4d59ee6e0088f91
>> The file SparkExamplesScript.scala, when pasted into spark-shell,
>> outputs the correct count() for the parallelized list comprehension,
>> as well as for the flatMapped RDD.
>>
>> The file SparkExamplesMinimal.scala contains exactly the same code,
>> and also the MASTER configuration and the Spark Executor are the same.
>> However, while the count() for the parallelized list is displayed
>> correctly, I receive the following error when asking for the count()
>> of the flatMapped RDD:
>>
>> -----------------
>>
>> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting Stage 1
>> (FlatMappedRDD[1] at flatMap at SparkExamplesMinimal.scala:34), which
>> has no missing parents
>> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting 8 missing
>> tasks from Stage 1 (FlatMappedRDD[1] at flatMap at
>> SparkExamplesMinimal.scala:34)
>> 14/05/21 09:47:49 INFO scheduler.TaskSchedulerImpl: Adding task set
>> 1.0 with 8 tasks
>> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Starting task 1.0:0
>> as TID 8 on executor 20140520-102159-2154735808-5050-1108-1: mesos9-1
>> (PROCESS_LOCAL)
>> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Serialized task 1.0:0
>> as 1779147 bytes in 37 ms
>> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Lost TID 8 (task 1.0:0)
>> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Loss was due to
>> java.lang.ClassNotFoundException
>> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60)
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
>> at
>> org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
>> at
>> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
>> at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> -----------------
>>
>> Can anyone explain to me where this comes from or how I might further
>> track the problem down?
>>
>> Thanks,
>> Tobias
>>
>
>
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Gerard Maas <ge...@gmail.com>.
Hi Tobias,
Regarding my comment on closure serialization:
I was discussing it with my fellow Sparkers here and I totally overlooked
the fact that you need the class files to de-serialize the closures (or
whatever) on the workers, so you always need the jar file delivered to the
workers in order for it to work.
The SparkREPL works differently. It uses some dark magic to send the
working session to the workers.
-kr, Gerard.
On Wed, May 21, 2014 at 2:47 PM, Gerard Maas <ge...@gmail.com> wrote:
> Hi Tobias,
>
> I was curious about this issue and tried to run your example on my local
> Mesos. I was able to reproduce your issue using your current config:
>
> [error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
> 1.0:4 failed 4 times (most recent failure: Exception failure:
> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2)
> org.apache.spark.SparkException: Job aborted: Task 1.0:4 failed 4 times
> (most recent failure: Exception failure: java.lang.ClassNotFoundException:
> spark.SparkExamplesMinimal$$anonfun$2)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
>
> Creating a simple jar from the job and providing it through the
> configuration seems to solve it:
>
> val conf = new SparkConf()
> .setMaster("mesos://<my_ip>:5050/")
> *
> .setJars(Seq("/sparkexample/target/scala-2.10/sparkexample_2.10-0.1.jar"))*
> .setAppName("SparkExamplesMinimal")
>
> Resulting in:
> 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Completed ResultTask(1, 1)
> 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Stage 1 (count at
> SparkExamplesMinimal.scala:50) finished in 1.120 s
> 14/05/21 12:03:45 INFO spark.SparkContext: Job finished: count at
> SparkExamplesMinimal.scala:50, took 1.177091435 s
> count: 1000000
>
> Why the closure serialization does not work with Mesos is beyond my
> current knowledge.
> Would be great to hear from the experts (cross-posting to dev for that)
>
> -kr, Gerard.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, May 21, 2014 at 11:51 AM, Tobias Pfeiffer <tg...@preferred.jp>wrote:
>
>> Hi,
>>
>> I have set up a cluster with Mesos (backed by Zookeeper) with three
>> master and three slave instances. I set up Spark (git HEAD) for use
>> with Mesos according to this manual:
>> http://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
>>
>> Using the spark-shell, I can connect to this cluster and do simple RDD
>> operations, but the same code in a Scala class and executed via sbt
>> run-main works only partially. (That is, count() works, count() after
>> flatMap() does not.)
>>
>> Here is my code: https://gist.github.com/tgpfeiffer/7d20a4d59ee6e0088f91
>> The file SparkExamplesScript.scala, when pasted into spark-shell,
>> outputs the correct count() for the parallelized list comprehension,
>> as well as for the flatMapped RDD.
>>
>> The file SparkExamplesMinimal.scala contains exactly the same code,
>> and also the MASTER configuration and the Spark Executor are the same.
>> However, while the count() for the parallelized list is displayed
>> correctly, I receive the following error when asking for the count()
>> of the flatMapped RDD:
>>
>> -----------------
>>
>> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting Stage 1
>> (FlatMappedRDD[1] at flatMap at SparkExamplesMinimal.scala:34), which
>> has no missing parents
>> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting 8 missing
>> tasks from Stage 1 (FlatMappedRDD[1] at flatMap at
>> SparkExamplesMinimal.scala:34)
>> 14/05/21 09:47:49 INFO scheduler.TaskSchedulerImpl: Adding task set
>> 1.0 with 8 tasks
>> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Starting task 1.0:0
>> as TID 8 on executor 20140520-102159-2154735808-5050-1108-1: mesos9-1
>> (PROCESS_LOCAL)
>> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Serialized task 1.0:0
>> as 1779147 bytes in 37 ms
>> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Lost TID 8 (task 1.0:0)
>> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Loss was due to
>> java.lang.ClassNotFoundException
>> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60)
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
>> at
>> org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
>> at
>> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
>> at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> -----------------
>>
>> Can anyone explain to me where this comes from or how I might further
>> track the problem down?
>>
>> Thanks,
>> Tobias
>>
>
>
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Gerard Maas <ge...@gmail.com>.
Hi Tobias,
For your simple example, I just used sbt package, but for more complex jobs
that have external dependencies, either:
- you should use sbt assembly [1] or mvn shade plugin [2] to build a "fat
jar" (aka jar-with-dependencies)
- or provide a list of jars including your job jar along with any
additional dependency: (e.g.
.setJars(Seq("sparkexample_2.10-0.1.jar","dependency1.jar",
"dependency2.jar", ... ,"dependencyN.jar"))
-kr, Gerard.
[1] https://github.com/sbt/sbt-assembly
[2] http://maven.apache.org/plugins/maven-shade-plugin
On Wed, May 21, 2014 at 4:04 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Gerard,
>
> thanks very much for your investigation! After hours of trial and
> error, I am kind of happy to hear it is not just a broken setup on my
> side that's causing the error.
>
> Could you explain briefly how you created that simple jar file?
>
> Thanks,
> Tobias
>
> On Wed, May 21, 2014 at 9:47 PM, Gerard Maas <ge...@gmail.com>
> wrote:
> > Hi Tobias,
> >
> > I was curious about this issue and tried to run your example on my local
> > Mesos. I was able to reproduce your issue using your current config:
> >
> > [error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
> > 1.0:4 failed 4 times (most recent failure: Exception failure:
> > java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2)
> > org.apache.spark.SparkException: Job aborted: Task 1.0:4 failed 4 times
> > (most recent failure: Exception failure:
> java.lang.ClassNotFoundException:
> > spark.SparkExamplesMinimal$$anonfun$2)
> > at
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
> >
> > Creating a simple jar from the job and providing it through the
> > configuration seems to solve it:
> >
> > val conf = new SparkConf()
> > .setMaster("mesos://<my_ip>:5050/")
> >
> >
> .setJars(Seq("/sparkexample/target/scala-2.10/sparkexample_2.10-0.1.jar"))
> > .setAppName("SparkExamplesMinimal")
> >
> > Resulting in:
> > 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Completed ResultTask(1, 1)
> > 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Stage 1 (count at
> > SparkExamplesMinimal.scala:50) finished in 1.120 s
> > 14/05/21 12:03:45 INFO spark.SparkContext: Job finished: count at
> > SparkExamplesMinimal.scala:50, took 1.177091435 s
> > count: 1000000
> >
> > Why the closure serialization does not work with Mesos is beyond my
> current
> > knowledge.
> > Would be great to hear from the experts (cross-posting to dev for that)
> >
> > -kr, Gerard.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, May 21, 2014 at 11:51 AM, Tobias Pfeiffer <tg...@preferred.jp>
> wrote:
> >>
> >> Hi,
> >>
> >> I have set up a cluster with Mesos (backed by Zookeeper) with three
> >> master and three slave instances. I set up Spark (git HEAD) for use
> >> with Mesos according to this manual:
> >> http://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
> >>
> >> Using the spark-shell, I can connect to this cluster and do simple RDD
> >> operations, but the same code in a Scala class and executed via sbt
> >> run-main works only partially. (That is, count() works, count() after
> >> flatMap() does not.)
> >>
> >> Here is my code:
> https://gist.github.com/tgpfeiffer/7d20a4d59ee6e0088f91
> >> The file SparkExamplesScript.scala, when pasted into spark-shell,
> >> outputs the correct count() for the parallelized list comprehension,
> >> as well as for the flatMapped RDD.
> >>
> >> The file SparkExamplesMinimal.scala contains exactly the same code,
> >> and also the MASTER configuration and the Spark Executor are the same.
> >> However, while the count() for the parallelized list is displayed
> >> correctly, I receive the following error when asking for the count()
> >> of the flatMapped RDD:
> >>
> >> -----------------
> >>
> >> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting Stage 1
> >> (FlatMappedRDD[1] at flatMap at SparkExamplesMinimal.scala:34), which
> >> has no missing parents
> >> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting 8 missing
> >> tasks from Stage 1 (FlatMappedRDD[1] at flatMap at
> >> SparkExamplesMinimal.scala:34)
> >> 14/05/21 09:47:49 INFO scheduler.TaskSchedulerImpl: Adding task set
> >> 1.0 with 8 tasks
> >> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Starting task 1.0:0
> >> as TID 8 on executor 20140520-102159-2154735808-5050-1108-1: mesos9-1
> >> (PROCESS_LOCAL)
> >> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Serialized task 1.0:0
> >> as 1779147 bytes in 37 ms
> >> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Lost TID 8 (task 1.0:0)
> >> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Loss was due to
> >> java.lang.ClassNotFoundException
> >> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:270)
> >> at
> >>
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60)
> >> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
> >> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> >> at
> >>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> >> at
> >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> >> at
> >>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> >> at
> >>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> >> at
> >>
> org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
> >> at
> >> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
> >> at
> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
> >> at
> >>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
> >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> >> at
> >>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> >> at
> >>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> -----------------
> >>
> >> Can anyone explain to me where this comes from or how I might further
> >> track the problem down?
> >>
> >> Thanks,
> >> Tobias
> >
> >
>
Re: ClassNotFoundException with Spark/Mesos (spark-shell works fine)
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Gerard,
thanks very much for your investigation! After hours of trial and
error, I am kind of happy to hear it is not just a broken setup on my
side that's causing the error.
Could you explain briefly how you created that simple jar file?
Thanks,
Tobias
On Wed, May 21, 2014 at 9:47 PM, Gerard Maas <ge...@gmail.com> wrote:
> Hi Tobias,
>
> I was curious about this issue and tried to run your example on my local
> Mesos. I was able to reproduce your issue using your current config:
>
> [error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task
> 1.0:4 failed 4 times (most recent failure: Exception failure:
> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2)
> org.apache.spark.SparkException: Job aborted: Task 1.0:4 failed 4 times
> (most recent failure: Exception failure: java.lang.ClassNotFoundException:
> spark.SparkExamplesMinimal$$anonfun$2)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
>
> Creating a simple jar from the job and providing it through the
> configuration seems to solve it:
>
> val conf = new SparkConf()
> .setMaster("mesos://<my_ip>:5050/")
>
> .setJars(Seq("/sparkexample/target/scala-2.10/sparkexample_2.10-0.1.jar"))
> .setAppName("SparkExamplesMinimal")
>
> Resulting in:
> 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Completed ResultTask(1, 1)
> 14/05/21 12:03:45 INFO scheduler.DAGScheduler: Stage 1 (count at
> SparkExamplesMinimal.scala:50) finished in 1.120 s
> 14/05/21 12:03:45 INFO spark.SparkContext: Job finished: count at
> SparkExamplesMinimal.scala:50, took 1.177091435 s
> count: 1000000
>
> Why the closure serialization does not work with Mesos is beyond my current
> knowledge.
> Would be great to hear from the experts (cross-posting to dev for that)
>
> -kr, Gerard.
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, May 21, 2014 at 11:51 AM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
>>
>> Hi,
>>
>> I have set up a cluster with Mesos (backed by Zookeeper) with three
>> master and three slave instances. I set up Spark (git HEAD) for use
>> with Mesos according to this manual:
>> http://people.apache.org/~pwendell/catalyst-docs/running-on-mesos.html
>>
>> Using the spark-shell, I can connect to this cluster and do simple RDD
>> operations, but the same code in a Scala class and executed via sbt
>> run-main works only partially. (That is, count() works, count() after
>> flatMap() does not.)
>>
>> Here is my code: https://gist.github.com/tgpfeiffer/7d20a4d59ee6e0088f91
>> The file SparkExamplesScript.scala, when pasted into spark-shell,
>> outputs the correct count() for the parallelized list comprehension,
>> as well as for the flatMapped RDD.
>>
>> The file SparkExamplesMinimal.scala contains exactly the same code,
>> and also the MASTER configuration and the Spark Executor are the same.
>> However, while the count() for the parallelized list is displayed
>> correctly, I receive the following error when asking for the count()
>> of the flatMapped RDD:
>>
>> -----------------
>>
>> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting Stage 1
>> (FlatMappedRDD[1] at flatMap at SparkExamplesMinimal.scala:34), which
>> has no missing parents
>> 14/05/21 09:47:49 INFO scheduler.DAGScheduler: Submitting 8 missing
>> tasks from Stage 1 (FlatMappedRDD[1] at flatMap at
>> SparkExamplesMinimal.scala:34)
>> 14/05/21 09:47:49 INFO scheduler.TaskSchedulerImpl: Adding task set
>> 1.0 with 8 tasks
>> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Starting task 1.0:0
>> as TID 8 on executor 20140520-102159-2154735808-5050-1108-1: mesos9-1
>> (PROCESS_LOCAL)
>> 14/05/21 09:47:49 INFO scheduler.TaskSetManager: Serialized task 1.0:0
>> as 1779147 bytes in 37 ms
>> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Lost TID 8 (task 1.0:0)
>> 14/05/21 09:47:49 WARN scheduler.TaskSetManager: Loss was due to
>> java.lang.ClassNotFoundException
>> java.lang.ClassNotFoundException: spark.SparkExamplesMinimal$$anonfun$2
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:270)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60)
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
>> at
>> org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
>> at
>> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
>> at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> -----------------
>>
>> Can anyone explain to me where this comes from or how I might further
>> track the problem down?
>>
>> Thanks,
>> Tobias
>
>