You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ilaxes <il...@hotmail.com> on 2015/03/13 18:52:29 UTC
serialization stakeoverflow error during reduce on nested objects
Hi,
I'm working on a RDD of a tuple of objects which represent trees (Node
containing a hashmap of nodes). I'm trying to aggregate these trees over the
RDD.
Let's take an example, 2 graphs :
C - D - B - A - D - B - E
F - E - B - A - D - B - F
I'm spliting each graphs according to the vertex A resulting in :
(B(1, D(1, C(1,))) , D(1, B(1, E(1,)))
(B(1, E(1, F(1,))) , D(1, B(1, F(1,)))
And I want to aggregate both graph getting :
(B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,)))
Some graph are potentially large (+4000 vertex) but I'm not supposed to have
any cyclic references.
When I run my program I get this error :
java.lang.StackOverflowError
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
at
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
I've tried to increase the size of the stake and use the standard java
serializer but no effect.
Any hint of the reason of this error and ways to change my code to solve it
?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: serialization stakeoverflow error during reduce on nested objects
Posted by alexis GILLAIN <il...@hotmail.com>.
I haven't register my class in kryo but I dont think it would have such an
impact on the stack size.
I'm thinking of using graphx and I'm wondering how it serializes the graph
object as it can use kryo as serializer.
2015-03-14 6:22 GMT+01:00 Ted Yu <yu...@gmail.com>:
> Have you registered your class with kryo ?
>
> See core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
> and core/src/test/scala/org/apache/spark/SparkConfSuite.scala
>
> On Fri, Mar 13, 2015 at 10:52 AM, ilaxes <il...@hotmail.com> wrote:
>
>> Hi,
>>
>> I'm working on a RDD of a tuple of objects which represent trees (Node
>> containing a hashmap of nodes). I'm trying to aggregate these trees over
>> the
>> RDD.
>>
>> Let's take an example, 2 graphs :
>> C - D - B - A - D - B - E
>> F - E - B - A - D - B - F
>>
>> I'm spliting each graphs according to the vertex A resulting in :
>> (B(1, D(1, C(1,))) , D(1, B(1, E(1,)))
>> (B(1, E(1, F(1,))) , D(1, B(1, F(1,)))
>>
>> And I want to aggregate both graph getting :
>> (B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,)))
>>
>> Some graph are potentially large (+4000 vertex) but I'm not supposed to
>> have
>> any cyclic references.
>> When I run my program I get this error :
>>
>> java.lang.StackOverflowError
>> at
>>
>> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127)
>> at
>>
>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
>> at
>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
>> at
>>
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
>> at
>>
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
>> at
>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
>> at
>>
>> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
>> at
>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
>> at
>>
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
>> at
>>
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
>>
>> I've tried to increase the size of the stake and use the standard java
>> serializer but no effect.
>>
>> Any hint of the reason of this error and ways to change my code to solve
>> it
>> ?
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: serialization stakeoverflow error during reduce on nested objects
Posted by Ted Yu <yu...@gmail.com>.
Have you registered your class with kryo ?
See core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
and core/src/test/scala/org/apache/spark/SparkConfSuite.scala
On Fri, Mar 13, 2015 at 10:52 AM, ilaxes <il...@hotmail.com> wrote:
> Hi,
>
> I'm working on a RDD of a tuple of objects which represent trees (Node
> containing a hashmap of nodes). I'm trying to aggregate these trees over
> the
> RDD.
>
> Let's take an example, 2 graphs :
> C - D - B - A - D - B - E
> F - E - B - A - D - B - F
>
> I'm spliting each graphs according to the vertex A resulting in :
> (B(1, D(1, C(1,))) , D(1, B(1, E(1,)))
> (B(1, E(1, F(1,))) , D(1, B(1, F(1,)))
>
> And I want to aggregate both graph getting :
> (B(2, (D(1, C(1,)), E(1, F(1,)))) , D(2, B(2, (E(1,), F(1,)))
>
> Some graph are potentially large (+4000 vertex) but I'm not supposed to
> have
> any cyclic references.
> When I run my program I get this error :
>
> java.lang.StackOverflowError
> at
>
> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:127)
> at
>
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721)
> at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
> at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
> at
>
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
> at
>
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
> at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
> at
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
>
> I've tried to increase the size of the stake and use the standard java
> serializer but no effect.
>
> Any hint of the reason of this error and ways to change my code to solve it
> ?
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/serialization-stakeoverflow-error-during-reduce-on-nested-objects-tp22040.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>