You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Hilmi Yildirim <Hi...@dfki.de> on 2016/01/21 10:47:51 UTC
Stackoverflow in Serialization
Hi,
I am using flink to train a ML model. There, I need to broadcast huge
broadcast variables in each iteration. During the serialization I get a
Stackoverflow exception. If I increase the Stack size then it works
well. The Stackoverflow could be avoided If you do not use recursion in
serialization. You could use iterations instead, I know it is more
complex but this would avoid a stack overflow.
Best Regards,
Hilmi
Re: Stackoverflow in Serialization
Posted by Till Rohrmann <tr...@apache.org>.
As you can see from the namespace and the object names, it's not our code
which causes the stack overflow. It's Kryo code which you're calling and
thus we have not control over it.
And in fact, I doubt that the problem is related to the number of elements
in your map but rather to the structure of the types. My guess is that you
have a type which contains a field of another type, which contains a field
of another type, and so on. You should check that.
Cheers,
Till
On Thu, Jan 21, 2016 at 11:26 AM, Hilmi Yildirim <Hi...@dfki.de>
wrote:
> Hi,
> I have a complex object with a MutableList and a Map containing thousands
> of objects. Here is a snippet of the exception. The actual exception is
> much longer. There, you can see that the exception is caused by too many
> recursive calls. For each recursive call the stack size increases because
> the parameters of each resursion level are saved on the stack. I am not
> sure but I suggest that the Map is the problem.
>
> Exception in thread "main" java.lang.StackOverflowError
> at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:445)
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
> at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
> at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>
>
>
> Best Regards,
> Hilmi
>
>
>
>
>
>
> Am 21.01.2016 um 11:14 schrieb Till Rohrmann:
>
>> Hi Hilmi,
>>
>> I've never seen such a problem. Could you please provide a bit more
>> background information. Which types are you serializing? Which serializer
>> are you using? What do you mean with recursion? Maybe you can provide a
>> simple example program which reproduces the problem.
>>
>> The thing is that if you have a composite data type, then each field will
>> be serialized with it's own serializer. Thus, if you data type is strongly
>> nested, then you will end up with a long chain of nested serializer calls.
>> In order to cause a stack overflow exception, you would have to have a
>> really deeply nested data structure. Maybe you can flatten it a bit.
>>
>> Cheers,
>> Till
>>
>> On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <Hi...@dfki.de>
>> wrote:
>>
>> Hi,
>>> I am using flink to train a ML model. There, I need to broadcast huge
>>> broadcast variables in each iteration. During the serialization I get a
>>> Stackoverflow exception. If I increase the Stack size then it works well.
>>> The Stackoverflow could be avoided If you do not use recursion in
>>> serialization. You could use iterations instead, I know it is more
>>> complex
>>> but this would avoid a stack overflow.
>>>
>>> Best Regards,
>>> Hilmi
>>>
>>>
> ==================================================================
> Hilmi Yildirim, M.Sc.
> Researcher
>
> DFKI GmbH
> Intelligente Analytik für Massendaten
> DFKI Projektbüro Berlin
> Alt-Moabit 91c
> D-10559 Berlin
> Phone: +49 30 23895 1814
>
> E-Mail: Hilmi.Yildirim@dfki.de <ma...@dfki.de>
>
> -------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>
> --
> ==================================================================
> Hilmi Yildirim, M.Sc.
> Researcher
>
> DFKI GmbH
> Intelligente Analytik für Massendaten
> DFKI Projektbüro Berlin
> Alt-Moabit 91c
> D-10559 Berlin
> Phone: +49 30 23895 1814
>
> E-Mail: Hilmi.Yildirim@dfki.de
>
> -------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>
>
Re: Stackoverflow in Serialization
Posted by Hilmi Yildirim <Hi...@dfki.de>.
Hi,
I have a complex object with a MutableList and a Map containing
thousands of objects. Here is a snippet of the exception. The actual
exception is much longer. There, you can see that the exception is
caused by too many recursive calls. For each recursive call the stack
size increases because the parameters of each resursion level are saved
on the stack. I am not sure but I suggest that the Map is the problem.
Exception in thread "main" java.lang.StackOverflowError
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:445)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
Best Regards,
Hilmi
Am 21.01.2016 um 11:14 schrieb Till Rohrmann:
> Hi Hilmi,
>
> I've never seen such a problem. Could you please provide a bit more
> background information. Which types are you serializing? Which serializer
> are you using? What do you mean with recursion? Maybe you can provide a
> simple example program which reproduces the problem.
>
> The thing is that if you have a composite data type, then each field will
> be serialized with it's own serializer. Thus, if you data type is strongly
> nested, then you will end up with a long chain of nested serializer calls.
> In order to cause a stack overflow exception, you would have to have a
> really deeply nested data structure. Maybe you can flatten it a bit.
>
> Cheers,
> Till
>
> On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <Hi...@dfki.de>
> wrote:
>
>> Hi,
>> I am using flink to train a ML model. There, I need to broadcast huge
>> broadcast variables in each iteration. During the serialization I get a
>> Stackoverflow exception. If I increase the Stack size then it works well.
>> The Stackoverflow could be avoided If you do not use recursion in
>> serialization. You could use iterations instead, I know it is more complex
>> but this would avoid a stack overflow.
>>
>> Best Regards,
>> Hilmi
>>
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: Hilmi.Yildirim@dfki.de <ma...@dfki.de>
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: Hilmi.Yildirim@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
Re: Stackoverflow in Serialization
Posted by Till Rohrmann <tr...@apache.org>.
Hi Hilmi,
I've never seen such a problem. Could you please provide a bit more
background information. Which types are you serializing? Which serializer
are you using? What do you mean with recursion? Maybe you can provide a
simple example program which reproduces the problem.
The thing is that if you have a composite data type, then each field will
be serialized with it's own serializer. Thus, if you data type is strongly
nested, then you will end up with a long chain of nested serializer calls.
In order to cause a stack overflow exception, you would have to have a
really deeply nested data structure. Maybe you can flatten it a bit.
Cheers,
Till
On Thu, Jan 21, 2016 at 10:47 AM, Hilmi Yildirim <Hi...@dfki.de>
wrote:
> Hi,
> I am using flink to train a ML model. There, I need to broadcast huge
> broadcast variables in each iteration. During the serialization I get a
> Stackoverflow exception. If I increase the Stack size then it works well.
> The Stackoverflow could be avoided If you do not use recursion in
> serialization. You could use iterations instead, I know it is more complex
> but this would avoid a stack overflow.
>
> Best Regards,
> Hilmi
>