You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Luis Mariano Guerra <ma...@event-fabric.com> on 2016/09/27 10:22:48 UTC
solution for this kryo serialization problem? (also: update to kryo 4.0.0?)
hi, I created an issue at kryo with a simple project that reproduces the
problem, but still asking here in case anyone knows a solution:
https://github.com/EsotericSoftware/kryo/issues/462
basically, I get an error trying to serialize a simple tree like structure,
the error happens with a 2 node, 1 leaf tree so it's not a recursion
problem caused by the size of the data.
anyone knows a way to avoid this error?
also, in the repo I tried with the 3 latest stable versions of kryo
(2.24.0, 3.0.3 an 4.0.0) and 4.0.0 seems to fix it, I tried building flink
and testing the update and with 3.0.3 it seems to work, but the build fails
for 4.0.0, didn't looked much into details since I hadn't much time.
I think at least moving to 3.0.3 would be a good thing in case this is
true: https://twitter.com/novoj/status/780693354610880514
Re: solution for this kryo serialization problem? (also: update to
kryo 4.0.0?)
Posted by Luis Mariano Guerra <ma...@event-fabric.com>.
I opened a PR on kryo https://github.com/EsotericSoftware/kryo/pull/463
that fixes 2.24.0, we should push for a 2.24.1 release, in the meantime if
you want to try the fix it's in this branch:
https://github.com/marianoguerra/kryo/tree/fix-462-for-2.24.0
I made a build of flink 1.1.2 with this fix and my problem went away.
On Tue, Sep 27, 2016 at 1:55 PM, Timo Walther <tw...@apache.org> wrote:
> Valid argument. I will add a comment to the issue.
>
>
> Am 27/09/16 um 14:25 schrieb Stephan Ewen:
>
> I would not bump the KRyo version easily - the serialization format
> changed (that's why they have a new major version), which would render all
> Flink savepoints and checkpoints incompatible.
>
> On Tue, Sep 27, 2016 at 1:19 PM, Timo Walther <tw...@apache.org> wrote:
>
>> Hi Luis,
>>
>> there is already an issue for bumping up the kryo version (
>> https://issues.apache.org/jira/browse/FLINK-3154). You could open a PR
>> if you like.
>>
>> Timo
>>
>> Am 27/09/16 um 13:10 schrieb Luis Mariano Guerra:
>>
>> On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra <
>> mariano@event-fabric.com> wrote:
>>
>>> hi, I created an issue at kryo with a simple project that reproduces the
>>> problem, but still asking here in case anyone knows a solution:
>>>
>>> https://github.com/EsotericSoftware/kryo/issues/462
>>>
>>
>> I found a solution to the problem, it's an extra if in Kryo that solves
>> the problem and can be applied to 2.24.0, so if they accept a PR flink can
>> move to 2.24.1 and not 4.0.0 :) (still the tweet I mention may be a good
>> reason to move to 3.0.4)
>>
>>
>>>
>>>
>>> basically, I get an error trying to serialize a simple tree like
>>> structure, the error happens with a 2 node, 1 leaf tree so it's not a
>>> recursion problem caused by the size of the data.
>>>
>>> anyone knows a way to avoid this error?
>>>
>>> also, in the repo I tried with the 3 latest stable versions of kryo
>>> (2.24.0, 3.0.3 an 4.0.0) and 4.0.0 seems to fix it, I tried building flink
>>> and testing the update and with 3.0.3 it seems to work, but the build fails
>>> for 4.0.0, didn't looked much into details since I hadn't much time.
>>>
>>> I think at least moving to 3.0.3 would be a good thing in case this is
>>> true: https://twitter.com/novoj/status/780693354610880514
>>>
>>>
>>
>>
>> --
>> Freundliche Grüße / Kind Regards
>>
>> Timo Walther
>>
>> Follow me: @twalthrhttps://www.linkedin.com/in/twalthr
>>
>> --
> Freundliche Grüße / Kind Regards
>
> Timo Walther
>
> Follow me: @twalthrhttps://www.linkedin.com/in/twalthr
>
>
Re: solution for this kryo serialization problem? (also: update to
kryo 4.0.0?)
Posted by Timo Walther <tw...@apache.org>.
Valid argument. I will add a comment to the issue.
Am 27/09/16 um 14:25 schrieb Stephan Ewen:
> I would not bump the KRyo version easily - the serialization format
> changed (that's why they have a new major version), which would render
> all Flink savepoints and checkpoints incompatible.
>
> On Tue, Sep 27, 2016 at 1:19 PM, Timo Walther <twalthr@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Luis,
>
> there is already an issue for bumping up the kryo version
> (https://issues.apache.org/jira/browse/FLINK-3154
> <https://issues.apache.org/jira/browse/FLINK-3154>). You could
> open a PR if you like.
>
> Timo
>
> Am 27/09/16 um 13:10 schrieb Luis Mariano Guerra:
>> On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra
>> <mariano@event-fabric.com <ma...@event-fabric.com>> wrote:
>>
>> hi, I created an issue at kryo with a simple project that
>> reproduces the problem, but still asking here in case anyone
>> knows a solution:
>>
>> https://github.com/EsotericSoftware/kryo/issues/462
>> <https://github.com/EsotericSoftware/kryo/issues/462>
>>
>>
>> I found a solution to the problem, it's an extra if in Kryo that
>> solves the problem and can be applied to 2.24.0, so if they
>> accept a PR flink can move to 2.24.1 and not 4.0.0 :) (still the
>> tweet I mention may be a good reason to move to 3.0.4)
>>
>>
>>
>> basically, I get an error trying to serialize a simple tree
>> like structure, the error happens with a 2 node, 1 leaf tree
>> so it's not a recursion problem caused by the size of the data.
>>
>> anyone knows a way to avoid this error?
>>
>> also, in the repo I tried with the 3 latest stable versions
>> of kryo (2.24.0, 3.0.3 an 4.0.0) and 4.0.0 seems to fix it, I
>> tried building flink and testing the update and with 3.0.3 it
>> seems to work, but the build fails for 4.0.0, didn't looked
>> much into details since I hadn't much time.
>>
>> I think at least moving to 3.0.3 would be a good thing in
>> case this is true:
>> https://twitter.com/novoj/status/780693354610880514
>> <https://twitter.com/novoj/status/780693354610880514>
>>
>>
>
>
> --
> Freundliche Gr��e / Kind Regards
>
> Timo Walther
>
> Follow me: @twalthr
> https://www.linkedin.com/in/twalthr
> <https://www.linkedin.com/in/twalthr>
>
--
Freundliche Gr��e / Kind Regards
Timo Walther
Follow me: @twalthr
https://www.linkedin.com/in/twalthr
Re: solution for this kryo serialization problem? (also: update to
kryo 4.0.0?)
Posted by Stephan Ewen <se...@apache.org>.
I would not bump the KRyo version easily - the serialization format changed
(that's why they have a new major version), which would render all Flink
savepoints and checkpoints incompatible.
On Tue, Sep 27, 2016 at 1:19 PM, Timo Walther <tw...@apache.org> wrote:
> Hi Luis,
>
> there is already an issue for bumping up the kryo version (
> https://issues.apache.org/jira/browse/FLINK-3154). You could open a PR if
> you like.
>
> Timo
>
> Am 27/09/16 um 13:10 schrieb Luis Mariano Guerra:
>
> On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra <
> mariano@event-fabric.com> wrote:
>
>> hi, I created an issue at kryo with a simple project that reproduces the
>> problem, but still asking here in case anyone knows a solution:
>>
>> https://github.com/EsotericSoftware/kryo/issues/462
>>
>
> I found a solution to the problem, it's an extra if in Kryo that solves
> the problem and can be applied to 2.24.0, so if they accept a PR flink can
> move to 2.24.1 and not 4.0.0 :) (still the tweet I mention may be a good
> reason to move to 3.0.4)
>
>
>>
>>
>> basically, I get an error trying to serialize a simple tree like
>> structure, the error happens with a 2 node, 1 leaf tree so it's not a
>> recursion problem caused by the size of the data.
>>
>> anyone knows a way to avoid this error?
>>
>> also, in the repo I tried with the 3 latest stable versions of kryo
>> (2.24.0, 3.0.3 an 4.0.0) and 4.0.0 seems to fix it, I tried building flink
>> and testing the update and with 3.0.3 it seems to work, but the build fails
>> for 4.0.0, didn't looked much into details since I hadn't much time.
>>
>> I think at least moving to 3.0.3 would be a good thing in case this is
>> true: https://twitter.com/novoj/status/780693354610880514
>>
>>
>
>
> --
> Freundliche Grüße / Kind Regards
>
> Timo Walther
>
> Follow me: @twalthrhttps://www.linkedin.com/in/twalthr
>
>
Re: solution for this kryo serialization problem? (also: update to
kryo 4.0.0?)
Posted by Timo Walther <tw...@apache.org>.
Hi Luis,
there is already an issue for bumping up the kryo version
(https://issues.apache.org/jira/browse/FLINK-3154). You could open a PR
if you like.
Timo
Am 27/09/16 um 13:10 schrieb Luis Mariano Guerra:
> On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra
> <mariano@event-fabric.com <ma...@event-fabric.com>> wrote:
>
> hi, I created an issue at kryo with a simple project that
> reproduces the problem, but still asking here in case anyone knows
> a solution:
>
> https://github.com/EsotericSoftware/kryo/issues/462
> <https://github.com/EsotericSoftware/kryo/issues/462>
>
>
> I found a solution to the problem, it's an extra if in Kryo that
> solves the problem and can be applied to 2.24.0, so if they accept a
> PR flink can move to 2.24.1 and not 4.0.0 :) (still the tweet I
> mention may be a good reason to move to 3.0.4)
>
>
>
> basically, I get an error trying to serialize a simple tree like
> structure, the error happens with a 2 node, 1 leaf tree so it's
> not a recursion problem caused by the size of the data.
>
> anyone knows a way to avoid this error?
>
> also, in the repo I tried with the 3 latest stable versions of
> kryo (2.24.0, 3.0.3 an 4.0.0) and 4.0.0 seems to fix it, I tried
> building flink and testing the update and with 3.0.3 it seems to
> work, but the build fails for 4.0.0, didn't looked much into
> details since I hadn't much time.
>
> I think at least moving to 3.0.3 would be a good thing in case
> this is true: https://twitter.com/novoj/status/780693354610880514
> <https://twitter.com/novoj/status/780693354610880514>
>
>
--
Freundliche Gr��e / Kind Regards
Timo Walther
Follow me: @twalthr
https://www.linkedin.com/in/twalthr
Re: solution for this kryo serialization problem? (also: update to
kryo 4.0.0?)
Posted by Luis Mariano Guerra <ma...@event-fabric.com>.
On Tue, Sep 27, 2016 at 11:22 AM, Luis Mariano Guerra <
mariano@event-fabric.com> wrote:
> hi, I created an issue at kryo with a simple project that reproduces the
> problem, but still asking here in case anyone knows a solution:
>
> https://github.com/EsotericSoftware/kryo/issues/462
>
I found a solution to the problem, it's an extra if in Kryo that solves the
problem and can be applied to 2.24.0, so if they accept a PR flink can move
to 2.24.1 and not 4.0.0 :) (still the tweet I mention may be a good reason
to move to 3.0.4)
>
>
> basically, I get an error trying to serialize a simple tree like
> structure, the error happens with a 2 node, 1 leaf tree so it's not a
> recursion problem caused by the size of the data.
>
> anyone knows a way to avoid this error?
>
> also, in the repo I tried with the 3 latest stable versions of kryo
> (2.24.0, 3.0.3 an 4.0.0) and 4.0.0 seems to fix it, I tried building flink
> and testing the update and with 3.0.3 it seems to work, but the build fails
> for 4.0.0, didn't looked much into details since I hadn't much time.
>
> I think at least moving to 3.0.3 would be a good thing in case this is
> true: https://twitter.com/novoj/status/780693354610880514
>
>