You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Teunissen, F.G.J. (Fred)" <Fr...@ing.com.INVALID> on 2021/01/15 10:40:58 UTC
outdated version of com.esotericsoftware.kryo
Hi,
While I was resolving a dependency version conflict in our project that uses Flink v1.11.2 I noticed that Flink is using the library ‘com.esotericsoftware.kryo:kryo<https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo>:2.24.0’.
Is there a reason why this library isn’t updated to a more recent version (the last one is `com.esotericsoftware: kryo<https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
Note. The groupId has been changed, so that could be the reason why it is missed.
Regards,
Fred Teunissen
-----------------------------------------------------------------
ATTENTION:
The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately.
-----------------------------------------------------------------
Re: outdated version of com.esotericsoftware.kryo
Posted by Seth Wiesman <sj...@gmail.com>.
To expand on Till's answer, Flink uses Kryo whenever the type system does
not recognize a given type. If your job works with Kryo fallback disabled,
there should be no usage of Kryo serialization in your snapshots[1]. In
theory, it should be possible to upgrade Kryo using Flink's schema
migration capabilities but I'm not aware of anyone who has looked into the
details or is working on this.
Seth
[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/types_serialization.html#disabling-kryo-fallback
On Fri, Jan 15, 2021 at 10:46 AM Till Rohrmann <tr...@apache.org> wrote:
> Yes, I think that Flink uses whatever key-/value-serializer is defined for
> serializing the keys/values for a savepoint.
>
> I haven't looked in detail into the problem but if Kryo offers some
> migration options, then this could make the problem a bit easier.
>
> Cheers,
> Till
>
>
>
> On Fri, Jan 15, 2021 at 2:44 PM Niels Basjes <Ni...@basjes.nl> wrote:
>
> > Till,
> >
> > So, we save the savepoints using the Kryo native format?
> > But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
> > that's ... really old !
> > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0
> >
> > I see that Kryo has some suggestions on the migration options:
> >
> >
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> > One thing is that they have an extra artifact that allows having multiple
> > versions of Kryo in a single application to make this possible.
> >
> > Niels Basjes
> >
> > On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> > > Hi Fred,
> > >
> > > one reason why this hasn't been updated is that Flink needs to ensure
> > > backwards compatibility. Hence, if the newer Kryo library serializes
> > > objects differently, then Flink would need to migrate old savepoints
> > having
> > > been stored via the old Kryo serializer to the new format. This is
> > > currently not supported and afaik blocks upgrading this library.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> > > <Fr...@ing.com.invalid> wrote:
> > >
> > > > Hi,
> > > >
> > > > While I was resolving a dependency version conflict in our project
> that
> > > > uses Flink v1.11.2 I noticed that Flink is using the library
> > > > ‘com.esotericsoftware.kryo:kryo<
> > > > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > > > >:2.24.0’.
> > > > Is there a reason why this library isn’t updated to a more recent
> > version
> > > > (the last one is `com.esotericsoftware: kryo<
> > > > https://mvnrepository.com/artifact/com.esotericsoftware/kryo
> >:5.0.3`)
> > > >
> > > > Note. The groupId has been changed, so that could be the reason why
> it
> > is
> > > > missed.
> > > >
> > > > Regards,
> > > > Fred Teunissen
> > > >
> > > > -----------------------------------------------------------------
> > > > ATTENTION:
> > > > The information in this e-mail is confidential and only meant for the
> > > > intended recipient. If you are not the intended recipient, don't use
> or
> > > > disclose it in any way. Please let the sender know and delete the
> > message
> > > > immediately.
> > > > -----------------------------------------------------------------
> > > >
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
Re: outdated version of com.esotericsoftware.kryo
Posted by Till Rohrmann <tr...@apache.org>.
Yes, I think that Flink uses whatever key-/value-serializer is defined for
serializing the keys/values for a savepoint.
I haven't looked in detail into the problem but if Kryo offers some
migration options, then this could make the problem a bit easier.
Cheers,
Till
On Fri, Jan 15, 2021 at 2:44 PM Niels Basjes <Ni...@basjes.nl> wrote:
> Till,
>
> So, we save the savepoints using the Kryo native format?
> But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
> that's ... really old !
> https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0
>
> I see that Kryo has some suggestions on the migration options:
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> One thing is that they have an extra artifact that allows having multiple
> versions of Kryo in a single application to make this possible.
>
> Niels Basjes
>
> On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
> > Hi Fred,
> >
> > one reason why this hasn't been updated is that Flink needs to ensure
> > backwards compatibility. Hence, if the newer Kryo library serializes
> > objects differently, then Flink would need to migrate old savepoints
> having
> > been stored via the old Kryo serializer to the new format. This is
> > currently not supported and afaik blocks upgrading this library.
> >
> > Cheers,
> > Till
> >
> > On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> > <Fr...@ing.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > While I was resolving a dependency version conflict in our project that
> > > uses Flink v1.11.2 I noticed that Flink is using the library
> > > ‘com.esotericsoftware.kryo:kryo<
> > > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > > >:2.24.0’.
> > > Is there a reason why this library isn’t updated to a more recent
> version
> > > (the last one is `com.esotericsoftware: kryo<
> > > https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
> > >
> > > Note. The groupId has been changed, so that could be the reason why it
> is
> > > missed.
> > >
> > > Regards,
> > > Fred Teunissen
> > >
> > > -----------------------------------------------------------------
> > > ATTENTION:
> > > The information in this e-mail is confidential and only meant for the
> > > intended recipient. If you are not the intended recipient, don't use or
> > > disclose it in any way. Please let the sender know and delete the
> message
> > > immediately.
> > > -----------------------------------------------------------------
> > >
> >
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
Re: outdated version of com.esotericsoftware.kryo
Posted by Niels Basjes <Ni...@basjes.nl>.
Till,
So, we save the savepoints using the Kryo native format?
But I see that Kryo 2.24 (as mentioned here) is from may 2014 ...
that's ... really old !
https://github.com/EsotericSoftware/kryo/releases/tag/kryo-2.24.0
I see that Kryo has some suggestions on the migration options:
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
One thing is that they have an extra artifact that allows having multiple
versions of Kryo in a single application to make this possible.
Niels Basjes
On Fri, Jan 15, 2021 at 2:14 PM Till Rohrmann <tr...@apache.org> wrote:
> Hi Fred,
>
> one reason why this hasn't been updated is that Flink needs to ensure
> backwards compatibility. Hence, if the newer Kryo library serializes
> objects differently, then Flink would need to migrate old savepoints having
> been stored via the old Kryo serializer to the new format. This is
> currently not supported and afaik blocks upgrading this library.
>
> Cheers,
> Till
>
> On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
> <Fr...@ing.com.invalid> wrote:
>
> > Hi,
> >
> > While I was resolving a dependency version conflict in our project that
> > uses Flink v1.11.2 I noticed that Flink is using the library
> > ‘com.esotericsoftware.kryo:kryo<
> > https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> > >:2.24.0’.
> > Is there a reason why this library isn’t updated to a more recent version
> > (the last one is `com.esotericsoftware: kryo<
> > https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
> >
> > Note. The groupId has been changed, so that could be the reason why it is
> > missed.
> >
> > Regards,
> > Fred Teunissen
> >
> > -----------------------------------------------------------------
> > ATTENTION:
> > The information in this e-mail is confidential and only meant for the
> > intended recipient. If you are not the intended recipient, don't use or
> > disclose it in any way. Please let the sender know and delete the message
> > immediately.
> > -----------------------------------------------------------------
> >
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: outdated version of com.esotericsoftware.kryo
Posted by Till Rohrmann <tr...@apache.org>.
Hi Fred,
one reason why this hasn't been updated is that Flink needs to ensure
backwards compatibility. Hence, if the newer Kryo library serializes
objects differently, then Flink would need to migrate old savepoints having
been stored via the old Kryo serializer to the new format. This is
currently not supported and afaik blocks upgrading this library.
Cheers,
Till
On Fri, Jan 15, 2021 at 11:41 AM Teunissen, F.G.J. (Fred)
<Fr...@ing.com.invalid> wrote:
> Hi,
>
> While I was resolving a dependency version conflict in our project that
> uses Flink v1.11.2 I noticed that Flink is using the library
> ‘com.esotericsoftware.kryo:kryo<
> https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo
> >:2.24.0’.
> Is there a reason why this library isn’t updated to a more recent version
> (the last one is `com.esotericsoftware: kryo<
> https://mvnrepository.com/artifact/com.esotericsoftware/kryo>:5.0.3`)
>
> Note. The groupId has been changed, so that could be the reason why it is
> missed.
>
> Regards,
> Fred Teunissen
>
> -----------------------------------------------------------------
> ATTENTION:
> The information in this e-mail is confidential and only meant for the
> intended recipient. If you are not the intended recipient, don't use or
> disclose it in any way. Please let the sender know and delete the message
> immediately.
> -----------------------------------------------------------------
>