You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2025/02/21 06:04:52 UTC
[DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Hey all!
I would like to rekindle this discussion as it seems that it has stalled
several times in the past and we are nearing the point in time where the
decision has to be made with regards to 2.0. (we are already a bit late but
nevermind)
There has been numerous requests and efforts to upgrade Kryo to better
support newer Java versions and Java native types. I think we can all agree
that this change is inevitable one way or another.
The latest JIRA for this seems to be:
https://issues.apache.org/jira/browse/FLINK-3154
There is even an open PR that accomplishes this (currently in a state
incompatible way) but based on the discussion it seems that with some extra
complexity compatibility can even be preserved by having both the old and
new Kryo versions active at the same time.
The main question here is whether state compatibility is important for 2.0
with this regard or we want to bite the bullet and make this upgrade once
and for all.
Cheers,
Gyula
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Kurt Ostfeld <ku...@proton.me.INVALID>.
The only reason to keep the old version of Kryo and not either upgrade or remove Kryo is if there is a backward compatibility advantage. If backward compatibility is breaking anyway, this seems an easy choice.
Java record support was the big motivation to this upgrade. From some simple tests I've done, the Flink 2.0 branch code seems to work with Java records where Flink 1.x versions did not. The Flink 1.x code was automatically using Kryo, and the old Kryo fails with Java records, the Flink 2.0 code doesn't seem to use Kryo by default.
However, you still don't want to pull a super old version of the Kryo library into the project. The Kryo code is still used widely throughout the Flink code base. The old Kryo 2.x is fundamentally incompatible with Java records and will fail if it gets called with a Java record or some other collection/class that has a nested reference to a Java record. Either none of that Kryo code is actively being used, in which case, it should be removed, or it is being used and will fail when used with Java records.
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Gyula Fóra <gy...@gmail.com>.
This has been merged now (both master and release-2.0).
Thank you all for your feedback!
Gyula
On Thu, Feb 27, 2025 at 3:41 AM Zakelly Lan <za...@gmail.com> wrote:
> Sry for the late reply, also +1 to have this in 2.0 given that we don't
> guarantee backwards compatibility, and it is already not compatible in many
> ways. Looking forward to this.
>
>
> Best,
> Zakelly
>
> On Thu, Feb 27, 2025 at 9:20 AM Xintong Song <to...@gmail.com>
> wrote:
>
> > Thank you both for the efforts. The progress sounds great.
> >
> > JFYI, there are still a few other blocker issues, and based on the
> progress
> > of resolving them my estimation would be creating RC1 around next Friday.
> > So for this feature there's no need to rush. Take your time for code
> > reviewing and testing.
> >
> > As for the migration tool, I agree it's a bigger scope issue, and should
> > not block the release. Actually, what I have in mind is something
> > independent from any Flink version, and may serve for general
> incompatible
> > state migration.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Feb 27, 2025 at 4:16 AM Kurt Ostfeld
> <kurtostfeld@proton.me.invalid
> > >
> > wrote:
> >
> > > A state migration tool that can migrate savepoints/checkpoints is a
> great
> > > idea that would be very useful for a lot of scenarios. But that is a
> > bigger
> > > scope issue.
> > >
> > > (fyi, CI is passing and PR should be fully ready)
> > >
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Zakelly Lan <za...@gmail.com>.
Sry for the late reply, also +1 to have this in 2.0 given that we don't
guarantee backwards compatibility, and it is already not compatible in many
ways. Looking forward to this.
Best,
Zakelly
On Thu, Feb 27, 2025 at 9:20 AM Xintong Song <to...@gmail.com> wrote:
> Thank you both for the efforts. The progress sounds great.
>
> JFYI, there are still a few other blocker issues, and based on the progress
> of resolving them my estimation would be creating RC1 around next Friday.
> So for this feature there's no need to rush. Take your time for code
> reviewing and testing.
>
> As for the migration tool, I agree it's a bigger scope issue, and should
> not block the release. Actually, what I have in mind is something
> independent from any Flink version, and may serve for general incompatible
> state migration.
>
> Best,
>
> Xintong
>
>
>
> On Thu, Feb 27, 2025 at 4:16 AM Kurt Ostfeld <kurtostfeld@proton.me.invalid
> >
> wrote:
>
> > A state migration tool that can migrate savepoints/checkpoints is a great
> > idea that would be very useful for a lot of scenarios. But that is a
> bigger
> > scope issue.
> >
> > (fyi, CI is passing and PR should be fully ready)
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Xintong Song <to...@gmail.com>.
Thank you both for the efforts. The progress sounds great.
JFYI, there are still a few other blocker issues, and based on the progress
of resolving them my estimation would be creating RC1 around next Friday.
So for this feature there's no need to rush. Take your time for code
reviewing and testing.
As for the migration tool, I agree it's a bigger scope issue, and should
not block the release. Actually, what I have in mind is something
independent from any Flink version, and may serve for general incompatible
state migration.
Best,
Xintong
On Thu, Feb 27, 2025 at 4:16 AM Kurt Ostfeld <ku...@proton.me.invalid>
wrote:
> A state migration tool that can migrate savepoints/checkpoints is a great
> idea that would be very useful for a lot of scenarios. But that is a bigger
> scope issue.
>
> (fyi, CI is passing and PR should be fully ready)
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Kurt Ostfeld <ku...@proton.me.INVALID>.
A state migration tool that can migrate savepoints/checkpoints is a great idea that would be very useful for a lot of scenarios. But that is a bigger scope issue.
(fyi, CI is passing and PR should be fully ready)
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Xintong!
The PR is basically ready and CI almost passes (e2es work but there were
some minor checkstyle and related errors).
I will go through it and also test it manually a little today / tomorrow.
Unless something very unexpected comes up during the review and testing I
would expect to be able to merge this by Friday to the master.
Regarding the compatibility/migration tooling, I think this could be
possible based on what I have read so far but I am not completely sure how
the 2 different Kryo versions will interplay in the tooling.
Maybe Nick or Kurt can chime in here but I imagine we could move the
backward compatibility logic into this tool that was originally built into
the Kryo upgrade PR directly.
Cheers
Gyula
On Wed, Feb 26, 2025 at 1:07 PM Xintong Song <to...@gmail.com> wrote:
> Hi Gyula,
>
> Sorry for chime in late, just noticed the thread. And thanks for bringing
> this up.
>
> As one of the release managers, +1 for including this in Flink 2.0.
>
> - Flink 2.0 does not guarantee state compatibility. We've mentioned that
> in the 2.0-preview release announcement. In addition to the Scala
> upgrading
> as Martijn mentioned, we also introduced built-in serializer for
> collections types as default, which is also compatibility breaking.
> - Although the feature freeze date has already passed, I think there's a
> good reason to make this effort an exception. We've been preparing this
> release for ~2 years. Why not wait for another 1-2 weeks, if that really
> helps?
>
> IIRC, this has come up somewhere when we plan the release, and was not
> included only because no one was driving the effort. So it's really nice to
> see it being picked up.
>
> Just for managing the expectation, do we have an estimation on how long
> this may take?
>
> A related topic is, can we provide a state migration tool to help users
> migrate across incompatible checkpoints / savepoints? I.e., deserialize the
> states into in-memory objects with old serialization setups, and serialize
> them with new setups. Such a tool can be helpful not only in a
> state-incompatible major version bump, but also in scenarios such as:
>
> - The user wants to change the serialization setups
> - State incompatibility due to user logic change, which the user knows
> how to deal with but lacks methods to do it. E.g., adding a new field to
> the state data type, where a certain default value can be applied to the
> previous states.
>
> I'm not entirely sure whether this can work or not. Just lack the capacity
> to look into it.
>
> Best,
>
> Xintong
>
>
>
> On Wed, Feb 26, 2025 at 6:02 PM Gyula Fóra <gy...@gmail.com> wrote:
>
> > Thank you all for your feedback.
> >
> > Let's leave this open for another day and unless there is any negative
> > feedback we can go ahead with merging the PR to bump the version for 2.0
> >
> > Cheers
> > Gyula
> >
> > On Wed, Feb 26, 2025 at 10:56 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Thanks for bringing this to our attention! I would choose simplicity
> over
> > > backward compatibility given Flink 2.0 offers the opportunity for
> > breaking
> > > changes. We will benefit from it from long-term's perspective. +1 for
> > > upgrading Kryo in Flink 2.0 in a non compatible way.
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Wed, Feb 26, 2025 at 5:37 AM Nick Nezis <ni...@apache.org>
> wrote:
> > >
> > > > Thanks Martijn.
> > > >
> > > > That's really great context. In that case, then I'll change my
> previous
> > > > opinion. I agree that we should proceed with the simpler pull request
> > and
> > > > get it into the Flink 2.0 release.
> > > >
> > > > On 2025/02/25 14:06:20 Martijn Visser wrote:
> > > > > Hi all,
> > > > >
> > > > > For the record, I don't think we have a guarantee around backwards
> > > > > compatibility for Flink 2.0 anyway, given that we upgraded Scala to
> > the
> > > > > latest version (because of the bump to JDK 17) and that will
> > > potentially
> > > > > break savepoints when using Scala. So I think we should also put
> this
> > > in
> > > > > for Flink 2.0, and just have the right release notes/documentation
> > for
> > > > this.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Tue, Feb 25, 2025 at 3:31 AM Zhanghao Chen <
> > > zhanghao.chen@outlook.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Gyula,
> > > > > >
> > > > > > Thanks for bringing this up! Definitely +1 for upgrading Kryo in
> > > Flink
> > > > > > 2.0. As a side note, it might be useful to introduce customizable
> > > > generic
> > > > > > serializer support like Spark, where you can switch to your own
> > > > serializer
> > > > > > via the "spark.serializer" [1] option. Users starting new
> > > applications
> > > > can
> > > > > > introduce their own serialization stack in this case to resolve
> > Java
> > > > > > compatibility issue is this case or for other performance issues.
> > > > > >
> > > > > > [1] https://spark.apache.org/docs/latest/configuration.html
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > ________________________________
> > > > > > From: Gyula F?ra <gy...@gmail.com>
> > > > > > Sent: Friday, February 21, 2025 14:04
> > > > > > To: dev <de...@flink.apache.org>
> > > > > > Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
> > > > > >
> > > > > > Hey all!
> > > > > >
> > > > > > I would like to rekindle this discussion as it seems that it has
> > > > stalled
> > > > > > several times in the past and we are nearing the point in time
> > where
> > > > the
> > > > > > decision has to be made with regards to 2.0. (we are already a
> bit
> > > > late but
> > > > > > nevermind)
> > > > > >
> > > > > > There has been numerous requests and efforts to upgrade Kryo to
> > > better
> > > > > > support newer Java versions and Java native types. I think we can
> > all
> > > > agree
> > > > > > that this change is inevitable one way or another.
> > > > > >
> > > > > > The latest JIRA for this seems to be:
> > > > > > https://issues.apache.org/jira/browse/FLINK-3154
> > > > > >
> > > > > > There is even an open PR that accomplishes this (currently in a
> > state
> > > > > > incompatible way) but based on the discussion it seems that with
> > some
> > > > extra
> > > > > > complexity compatibility can even be preserved by having both the
> > old
> > > > and
> > > > > > new Kryo versions active at the same time.
> > > > > >
> > > > > > The main question here is whether state compatibility is
> important
> > > for
> > > > 2.0
> > > > > > with this regard or we want to bite the bullet and make this
> > upgrade
> > > > once
> > > > > > and for all.
> > > > > >
> > > > > > Cheers,
> > > > > > Gyula
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Xintong Song <to...@gmail.com>.
Hi Gyula,
Sorry for chime in late, just noticed the thread. And thanks for bringing
this up.
As one of the release managers, +1 for including this in Flink 2.0.
- Flink 2.0 does not guarantee state compatibility. We've mentioned that
in the 2.0-preview release announcement. In addition to the Scala upgrading
as Martijn mentioned, we also introduced built-in serializer for
collections types as default, which is also compatibility breaking.
- Although the feature freeze date has already passed, I think there's a
good reason to make this effort an exception. We've been preparing this
release for ~2 years. Why not wait for another 1-2 weeks, if that really
helps?
IIRC, this has come up somewhere when we plan the release, and was not
included only because no one was driving the effort. So it's really nice to
see it being picked up.
Just for managing the expectation, do we have an estimation on how long
this may take?
A related topic is, can we provide a state migration tool to help users
migrate across incompatible checkpoints / savepoints? I.e., deserialize the
states into in-memory objects with old serialization setups, and serialize
them with new setups. Such a tool can be helpful not only in a
state-incompatible major version bump, but also in scenarios such as:
- The user wants to change the serialization setups
- State incompatibility due to user logic change, which the user knows
how to deal with but lacks methods to do it. E.g., adding a new field to
the state data type, where a certain default value can be applied to the
previous states.
I'm not entirely sure whether this can work or not. Just lack the capacity
to look into it.
Best,
Xintong
On Wed, Feb 26, 2025 at 6:02 PM Gyula Fóra <gy...@gmail.com> wrote:
> Thank you all for your feedback.
>
> Let's leave this open for another day and unless there is any negative
> feedback we can go ahead with merging the PR to bump the version for 2.0
>
> Cheers
> Gyula
>
> On Wed, Feb 26, 2025 at 10:56 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Thanks for bringing this to our attention! I would choose simplicity over
> > backward compatibility given Flink 2.0 offers the opportunity for
> breaking
> > changes. We will benefit from it from long-term's perspective. +1 for
> > upgrading Kryo in Flink 2.0 in a non compatible way.
> >
> > Best regards,
> > Jing
> >
> > On Wed, Feb 26, 2025 at 5:37 AM Nick Nezis <ni...@apache.org> wrote:
> >
> > > Thanks Martijn.
> > >
> > > That's really great context. In that case, then I'll change my previous
> > > opinion. I agree that we should proceed with the simpler pull request
> and
> > > get it into the Flink 2.0 release.
> > >
> > > On 2025/02/25 14:06:20 Martijn Visser wrote:
> > > > Hi all,
> > > >
> > > > For the record, I don't think we have a guarantee around backwards
> > > > compatibility for Flink 2.0 anyway, given that we upgraded Scala to
> the
> > > > latest version (because of the bump to JDK 17) and that will
> > potentially
> > > > break savepoints when using Scala. So I think we should also put this
> > in
> > > > for Flink 2.0, and just have the right release notes/documentation
> for
> > > this.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Tue, Feb 25, 2025 at 3:31 AM Zhanghao Chen <
> > zhanghao.chen@outlook.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Gyula,
> > > > >
> > > > > Thanks for bringing this up! Definitely +1 for upgrading Kryo in
> > Flink
> > > > > 2.0. As a side note, it might be useful to introduce customizable
> > > generic
> > > > > serializer support like Spark, where you can switch to your own
> > > serializer
> > > > > via the "spark.serializer" [1] option. Users starting new
> > applications
> > > can
> > > > > introduce their own serialization stack in this case to resolve
> Java
> > > > > compatibility issue is this case or for other performance issues.
> > > > >
> > > > > [1] https://spark.apache.org/docs/latest/configuration.html
> > > > >
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: Gyula F?ra <gy...@gmail.com>
> > > > > Sent: Friday, February 21, 2025 14:04
> > > > > To: dev <de...@flink.apache.org>
> > > > > Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
> > > > >
> > > > > Hey all!
> > > > >
> > > > > I would like to rekindle this discussion as it seems that it has
> > > stalled
> > > > > several times in the past and we are nearing the point in time
> where
> > > the
> > > > > decision has to be made with regards to 2.0. (we are already a bit
> > > late but
> > > > > nevermind)
> > > > >
> > > > > There has been numerous requests and efforts to upgrade Kryo to
> > better
> > > > > support newer Java versions and Java native types. I think we can
> all
> > > agree
> > > > > that this change is inevitable one way or another.
> > > > >
> > > > > The latest JIRA for this seems to be:
> > > > > https://issues.apache.org/jira/browse/FLINK-3154
> > > > >
> > > > > There is even an open PR that accomplishes this (currently in a
> state
> > > > > incompatible way) but based on the discussion it seems that with
> some
> > > extra
> > > > > complexity compatibility can even be preserved by having both the
> old
> > > and
> > > > > new Kryo versions active at the same time.
> > > > >
> > > > > The main question here is whether state compatibility is important
> > for
> > > 2.0
> > > > > with this regard or we want to bite the bullet and make this
> upgrade
> > > once
> > > > > and for all.
> > > > >
> > > > > Cheers,
> > > > > Gyula
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Gyula Fóra <gy...@gmail.com>.
Thank you all for your feedback.
Let's leave this open for another day and unless there is any negative
feedback we can go ahead with merging the PR to bump the version for 2.0
Cheers
Gyula
On Wed, Feb 26, 2025 at 10:56 AM Jing Ge <ji...@ververica.com.invalid> wrote:
> Thanks for bringing this to our attention! I would choose simplicity over
> backward compatibility given Flink 2.0 offers the opportunity for breaking
> changes. We will benefit from it from long-term's perspective. +1 for
> upgrading Kryo in Flink 2.0 in a non compatible way.
>
> Best regards,
> Jing
>
> On Wed, Feb 26, 2025 at 5:37 AM Nick Nezis <ni...@apache.org> wrote:
>
> > Thanks Martijn.
> >
> > That's really great context. In that case, then I'll change my previous
> > opinion. I agree that we should proceed with the simpler pull request and
> > get it into the Flink 2.0 release.
> >
> > On 2025/02/25 14:06:20 Martijn Visser wrote:
> > > Hi all,
> > >
> > > For the record, I don't think we have a guarantee around backwards
> > > compatibility for Flink 2.0 anyway, given that we upgraded Scala to the
> > > latest version (because of the bump to JDK 17) and that will
> potentially
> > > break savepoints when using Scala. So I think we should also put this
> in
> > > for Flink 2.0, and just have the right release notes/documentation for
> > this.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Tue, Feb 25, 2025 at 3:31 AM Zhanghao Chen <
> zhanghao.chen@outlook.com
> > >
> > > wrote:
> > >
> > > > Hi Gyula,
> > > >
> > > > Thanks for bringing this up! Definitely +1 for upgrading Kryo in
> Flink
> > > > 2.0. As a side note, it might be useful to introduce customizable
> > generic
> > > > serializer support like Spark, where you can switch to your own
> > serializer
> > > > via the "spark.serializer" [1] option. Users starting new
> applications
> > can
> > > > introduce their own serialization stack in this case to resolve Java
> > > > compatibility issue is this case or for other performance issues.
> > > >
> > > > [1] https://spark.apache.org/docs/latest/configuration.html
> > > >
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: Gyula F?ra <gy...@gmail.com>
> > > > Sent: Friday, February 21, 2025 14:04
> > > > To: dev <de...@flink.apache.org>
> > > > Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
> > > >
> > > > Hey all!
> > > >
> > > > I would like to rekindle this discussion as it seems that it has
> > stalled
> > > > several times in the past and we are nearing the point in time where
> > the
> > > > decision has to be made with regards to 2.0. (we are already a bit
> > late but
> > > > nevermind)
> > > >
> > > > There has been numerous requests and efforts to upgrade Kryo to
> better
> > > > support newer Java versions and Java native types. I think we can all
> > agree
> > > > that this change is inevitable one way or another.
> > > >
> > > > The latest JIRA for this seems to be:
> > > > https://issues.apache.org/jira/browse/FLINK-3154
> > > >
> > > > There is even an open PR that accomplishes this (currently in a state
> > > > incompatible way) but based on the discussion it seems that with some
> > extra
> > > > complexity compatibility can even be preserved by having both the old
> > and
> > > > new Kryo versions active at the same time.
> > > >
> > > > The main question here is whether state compatibility is important
> for
> > 2.0
> > > > with this regard or we want to bite the bullet and make this upgrade
> > once
> > > > and for all.
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > >
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Thanks for bringing this to our attention! I would choose simplicity over
backward compatibility given Flink 2.0 offers the opportunity for breaking
changes. We will benefit from it from long-term's perspective. +1 for
upgrading Kryo in Flink 2.0 in a non compatible way.
Best regards,
Jing
On Wed, Feb 26, 2025 at 5:37 AM Nick Nezis <ni...@apache.org> wrote:
> Thanks Martijn.
>
> That's really great context. In that case, then I'll change my previous
> opinion. I agree that we should proceed with the simpler pull request and
> get it into the Flink 2.0 release.
>
> On 2025/02/25 14:06:20 Martijn Visser wrote:
> > Hi all,
> >
> > For the record, I don't think we have a guarantee around backwards
> > compatibility for Flink 2.0 anyway, given that we upgraded Scala to the
> > latest version (because of the bump to JDK 17) and that will potentially
> > break savepoints when using Scala. So I think we should also put this in
> > for Flink 2.0, and just have the right release notes/documentation for
> this.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Tue, Feb 25, 2025 at 3:31 AM Zhanghao Chen <zhanghao.chen@outlook.com
> >
> > wrote:
> >
> > > Hi Gyula,
> > >
> > > Thanks for bringing this up! Definitely +1 for upgrading Kryo in Flink
> > > 2.0. As a side note, it might be useful to introduce customizable
> generic
> > > serializer support like Spark, where you can switch to your own
> serializer
> > > via the "spark.serializer" [1] option. Users starting new applications
> can
> > > introduce their own serialization stack in this case to resolve Java
> > > compatibility issue is this case or for other performance issues.
> > >
> > > [1] https://spark.apache.org/docs/latest/configuration.html
> > >
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: Gyula F?ra <gy...@gmail.com>
> > > Sent: Friday, February 21, 2025 14:04
> > > To: dev <de...@flink.apache.org>
> > > Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
> > >
> > > Hey all!
> > >
> > > I would like to rekindle this discussion as it seems that it has
> stalled
> > > several times in the past and we are nearing the point in time where
> the
> > > decision has to be made with regards to 2.0. (we are already a bit
> late but
> > > nevermind)
> > >
> > > There has been numerous requests and efforts to upgrade Kryo to better
> > > support newer Java versions and Java native types. I think we can all
> agree
> > > that this change is inevitable one way or another.
> > >
> > > The latest JIRA for this seems to be:
> > > https://issues.apache.org/jira/browse/FLINK-3154
> > >
> > > There is even an open PR that accomplishes this (currently in a state
> > > incompatible way) but based on the discussion it seems that with some
> extra
> > > complexity compatibility can even be preserved by having both the old
> and
> > > new Kryo versions active at the same time.
> > >
> > > The main question here is whether state compatibility is important for
> 2.0
> > > with this regard or we want to bite the bullet and make this upgrade
> once
> > > and for all.
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Nick Nezis <ni...@apache.org>.
Thanks Martijn.
That's really great context. In that case, then I'll change my previous opinion. I agree that we should proceed with the simpler pull request and get it into the Flink 2.0 release.
On 2025/02/25 14:06:20 Martijn Visser wrote:
> Hi all,
>
> For the record, I don't think we have a guarantee around backwards
> compatibility for Flink 2.0 anyway, given that we upgraded Scala to the
> latest version (because of the bump to JDK 17) and that will potentially
> break savepoints when using Scala. So I think we should also put this in
> for Flink 2.0, and just have the right release notes/documentation for this.
>
> Best regards,
>
> Martijn
>
> On Tue, Feb 25, 2025 at 3:31 AM Zhanghao Chen <zh...@outlook.com>
> wrote:
>
> > Hi Gyula,
> >
> > Thanks for bringing this up! Definitely +1 for upgrading Kryo in Flink
> > 2.0. As a side note, it might be useful to introduce customizable generic
> > serializer support like Spark, where you can switch to your own serializer
> > via the "spark.serializer" [1] option. Users starting new applications can
> > introduce their own serialization stack in this case to resolve Java
> > compatibility issue is this case or for other performance issues.
> >
> > [1] https://spark.apache.org/docs/latest/configuration.html
> >
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: Gyula F?ra <gy...@gmail.com>
> > Sent: Friday, February 21, 2025 14:04
> > To: dev <de...@flink.apache.org>
> > Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
> >
> > Hey all!
> >
> > I would like to rekindle this discussion as it seems that it has stalled
> > several times in the past and we are nearing the point in time where the
> > decision has to be made with regards to 2.0. (we are already a bit late but
> > nevermind)
> >
> > There has been numerous requests and efforts to upgrade Kryo to better
> > support newer Java versions and Java native types. I think we can all agree
> > that this change is inevitable one way or another.
> >
> > The latest JIRA for this seems to be:
> > https://issues.apache.org/jira/browse/FLINK-3154
> >
> > There is even an open PR that accomplishes this (currently in a state
> > incompatible way) but based on the discussion it seems that with some extra
> > complexity compatibility can even be preserved by having both the old and
> > new Kryo versions active at the same time.
> >
> > The main question here is whether state compatibility is important for 2.0
> > with this regard or we want to bite the bullet and make this upgrade once
> > and for all.
> >
> > Cheers,
> > Gyula
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Martijn Visser <ma...@apache.org>.
Hi all,
For the record, I don't think we have a guarantee around backwards
compatibility for Flink 2.0 anyway, given that we upgraded Scala to the
latest version (because of the bump to JDK 17) and that will potentially
break savepoints when using Scala. So I think we should also put this in
for Flink 2.0, and just have the right release notes/documentation for this.
Best regards,
Martijn
On Tue, Feb 25, 2025 at 3:31 AM Zhanghao Chen <zh...@outlook.com>
wrote:
> Hi Gyula,
>
> Thanks for bringing this up! Definitely +1 for upgrading Kryo in Flink
> 2.0. As a side note, it might be useful to introduce customizable generic
> serializer support like Spark, where you can switch to your own serializer
> via the "spark.serializer" [1] option. Users starting new applications can
> introduce their own serialization stack in this case to resolve Java
> compatibility issue is this case or for other performance issues.
>
> [1] https://spark.apache.org/docs/latest/configuration.html
>
>
> Best,
> Zhanghao Chen
> ________________________________
> From: Gyula F?ra <gy...@gmail.com>
> Sent: Friday, February 21, 2025 14:04
> To: dev <de...@flink.apache.org>
> Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
>
> Hey all!
>
> I would like to rekindle this discussion as it seems that it has stalled
> several times in the past and we are nearing the point in time where the
> decision has to be made with regards to 2.0. (we are already a bit late but
> nevermind)
>
> There has been numerous requests and efforts to upgrade Kryo to better
> support newer Java versions and Java native types. I think we can all agree
> that this change is inevitable one way or another.
>
> The latest JIRA for this seems to be:
> https://issues.apache.org/jira/browse/FLINK-3154
>
> There is even an open PR that accomplishes this (currently in a state
> incompatible way) but based on the discussion it seems that with some extra
> complexity compatibility can even be preserved by having both the old and
> new Kryo versions active at the same time.
>
> The main question here is whether state compatibility is important for 2.0
> with this regard or we want to bite the bullet and make this upgrade once
> and for all.
>
> Cheers,
> Gyula
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Zhanghao Chen <zh...@outlook.com>.
Hi Gyula,
Thanks for bringing this up! Definitely +1 for upgrading Kryo in Flink 2.0. As a side note, it might be useful to introduce customizable generic serializer support like Spark, where you can switch to your own serializer via the "spark.serializer" [1] option. Users starting new applications can introduce their own serialization stack in this case to resolve Java compatibility issue is this case or for other performance issues.
[1] https://spark.apache.org/docs/latest/configuration.html
Best,
Zhanghao Chen
________________________________
From: Gyula F?ra <gy...@gmail.com>
Sent: Friday, February 21, 2025 14:04
To: dev <de...@flink.apache.org>
Subject: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Hey all!
I would like to rekindle this discussion as it seems that it has stalled
several times in the past and we are nearing the point in time where the
decision has to be made with regards to 2.0. (we are already a bit late but
nevermind)
There has been numerous requests and efforts to upgrade Kryo to better
support newer Java versions and Java native types. I think we can all agree
that this change is inevitable one way or another.
The latest JIRA for this seems to be:
https://issues.apache.org/jira/browse/FLINK-3154
There is even an open PR that accomplishes this (currently in a state
incompatible way) but based on the discussion it seems that with some extra
complexity compatibility can even be preserved by having both the old and
new Kryo versions active at the same time.
The main question here is whether state compatibility is important for 2.0
with this regard or we want to bite the bullet and make this upgrade once
and for all.
Cheers,
Gyula
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Nick Nezis <ni...@apache.org>.
Perhaps we should provide the Backward compatible version in the initial 2.0 release, and provided a deprecation plan for simplifying the code and only supporting the newer Kryo 5.x in Flink 2.1+. This would be inline with the previous decision provided in the FLIP's compatibility and deprecation section: https://cwiki.apache.org/confluence/display/FLINK/FLIP-317:+Upgrade+Kryo+from+2.24.0+to+5.5.0
From the FLIP:
"All Flink applications currently generate state with Kryo v2. This PR will fully support reading that state and write newer state with Kryo v5. However, some future version of Flink would eventually drop support for Kryo v2 and users will need to make sure all persisted state is upgraded before upgrading to a future version of Flink that drops support for Kryo v2 serialized state. There will be a migration pathway so that all applications can upgrade without losing state. Applications will have to run on a bridge release version that will read their state with Kryo v2 data and write it with Kryo v5 data before upgrading to a future version of Flink that completely drops support for Kryo v2."
Another distinction between PR #1 and PR #2 relates to the Twitter Chill dependency. I would recommend we still remove the twitter/chill dependency. This project is no longer maintained and we were not able to get our Kryo 5.x update merged in.
Nick
On 2025/02/24 11:58:35 Gyula Fóra wrote:
> Thanks for chiming in Timo, I completely agree.
>
> There seem to be 2 approaches available here:
> 1. Backward compatible: https://github.com/apache/flink/pull/22660
> 2. Non compatible https://github.com/apache/flink/pull/25896
>
> They both seem to be feasible however the backward compatible approach is
> much more complex overall. I don't really have experience with the Kryo
> logic to make an informed call here.
> Would be good to hear from someone with deeper experience with our Kryo
> serialization stack.
>
> Gyula
>
> On Mon, Feb 24, 2025 at 12:37 PM Timo Walther <tw...@apache.org> wrote:
>
> > Hi Gyula,
> >
> > thanks for bringing this up. I agree that we should use one of the most
> > recent Kryo versions for Flink 2.0. Otherwise the community will suffer
> > again from incompatibilities and needs to wait for another major Flink
> > version. Thanks for starting this discussion. We should do it for 2.0.
> > If not now, when?
> >
> > Cheers,
> > Timo
> >
> > On 24.02.25 08:03, Gyula Fóra wrote:
> > > Hey!
> > >
> > > I think the main question here is whether Flink 2.0 is going to be state
> > > backward compatible or not.
> > > If not then we need to make the upgrade right now, freeze or not. We have
> > > to decide this as a community.
> > >
> > > If we need to preserve backward compatibility then we need to go with a
> > > much more complex solution here when upgrading Kryo to do that.
> > >
> > > Do we have a community decision somewhere related to state
> > > backward compatibility for 2.0?
> > >
> > > Cheers
> > > Gyula
> > >
> > > On Mon, Feb 24, 2025 at 3:02 AM Rui Fan <19...@gmail.com> wrote:
> > >
> > >> Thanks Gyula for driving this discussion!
> > >>
> > >> +1 for upgrading kyro, I have a question about the timeline.
> > >> The flink 2.0.0 has been freezed, do we have enough time to
> > >> test if it's done in flink 2.0.0?
> > >>
> > >> Best,
> > >> Rui
> > >>
> > >> On Fri, Feb 21, 2025 at 9:55 PM Alexander Fedulov <af...@apache.org>
> > >> wrote:
> > >>
> > >>> Hi Gyula,
> > >>>
> > >>> Thanks for bringing up this topic! Kryo incompatibility with newer
> > >>> Java versions is a major issue that needs to be addressed, and in my
> > >>> opinion, Flink 2.0 provides a great opportunity to introduce this
> > >>> change.
> > >>>
> > >>> My understanding is that state compatibility was not a strict goal
> > >>> during 2.0 development. However, I’ve heard mentions that it might
> > >>> actually be compatible after all.
> > >>>
> > >>> From this perspective:
> > >>> definitely +1 on upgrading Kryo. The decision to invest additional
> > >>> effort in maintaining compatibility should depend on whether all other
> > >>> changes have preserved compatibility guarantees. If Kryo is the only
> > >>> breaking change, then ensuring compatibility might be worth
> > >>> considering.
> > >>>
> > >>> Best,
> > >>> Alex
> > >>>
> > >>> On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gy...@gmail.com> wrote:
> > >>>>
> > >>>> Hey all!
> > >>>>
> > >>>> I would like to rekindle this discussion as it seems that it has
> > >> stalled
> > >>>> several times in the past and we are nearing the point in time where
> > >> the
> > >>>> decision has to be made with regards to 2.0. (we are already a bit
> > late
> > >>> but
> > >>>> nevermind)
> > >>>>
> > >>>> There has been numerous requests and efforts to upgrade Kryo to better
> > >>>> support newer Java versions and Java native types. I think we can all
> > >>> agree
> > >>>> that this change is inevitable one way or another.
> > >>>>
> > >>>> The latest JIRA for this seems to be:
> > >>>> https://issues.apache.org/jira/browse/FLINK-3154
> > >>>>
> > >>>> There is even an open PR that accomplishes this (currently in a state
> > >>>> incompatible way) but based on the discussion it seems that with some
> > >>> extra
> > >>>> complexity compatibility can even be preserved by having both the old
> > >> and
> > >>>> new Kryo versions active at the same time.
> > >>>>
> > >>>> The main question here is whether state compatibility is important for
> > >>> 2.0
> > >>>> with this regard or we want to bite the bullet and make this upgrade
> > >> once
> > >>>> and for all.
> > >>>>
> > >>>> Cheers,
> > >>>> Gyula
> > >>>
> > >>
> > >
> >
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Gyula Fóra <gy...@gmail.com>.
Thanks for chiming in Timo, I completely agree.
There seem to be 2 approaches available here:
1. Backward compatible: https://github.com/apache/flink/pull/22660
2. Non compatible https://github.com/apache/flink/pull/25896
They both seem to be feasible however the backward compatible approach is
much more complex overall. I don't really have experience with the Kryo
logic to make an informed call here.
Would be good to hear from someone with deeper experience with our Kryo
serialization stack.
Gyula
On Mon, Feb 24, 2025 at 12:37 PM Timo Walther <tw...@apache.org> wrote:
> Hi Gyula,
>
> thanks for bringing this up. I agree that we should use one of the most
> recent Kryo versions for Flink 2.0. Otherwise the community will suffer
> again from incompatibilities and needs to wait for another major Flink
> version. Thanks for starting this discussion. We should do it for 2.0.
> If not now, when?
>
> Cheers,
> Timo
>
> On 24.02.25 08:03, Gyula Fóra wrote:
> > Hey!
> >
> > I think the main question here is whether Flink 2.0 is going to be state
> > backward compatible or not.
> > If not then we need to make the upgrade right now, freeze or not. We have
> > to decide this as a community.
> >
> > If we need to preserve backward compatibility then we need to go with a
> > much more complex solution here when upgrading Kryo to do that.
> >
> > Do we have a community decision somewhere related to state
> > backward compatibility for 2.0?
> >
> > Cheers
> > Gyula
> >
> > On Mon, Feb 24, 2025 at 3:02 AM Rui Fan <19...@gmail.com> wrote:
> >
> >> Thanks Gyula for driving this discussion!
> >>
> >> +1 for upgrading kyro, I have a question about the timeline.
> >> The flink 2.0.0 has been freezed, do we have enough time to
> >> test if it's done in flink 2.0.0?
> >>
> >> Best,
> >> Rui
> >>
> >> On Fri, Feb 21, 2025 at 9:55 PM Alexander Fedulov <af...@apache.org>
> >> wrote:
> >>
> >>> Hi Gyula,
> >>>
> >>> Thanks for bringing up this topic! Kryo incompatibility with newer
> >>> Java versions is a major issue that needs to be addressed, and in my
> >>> opinion, Flink 2.0 provides a great opportunity to introduce this
> >>> change.
> >>>
> >>> My understanding is that state compatibility was not a strict goal
> >>> during 2.0 development. However, I’ve heard mentions that it might
> >>> actually be compatible after all.
> >>>
> >>> From this perspective:
> >>> definitely +1 on upgrading Kryo. The decision to invest additional
> >>> effort in maintaining compatibility should depend on whether all other
> >>> changes have preserved compatibility guarantees. If Kryo is the only
> >>> breaking change, then ensuring compatibility might be worth
> >>> considering.
> >>>
> >>> Best,
> >>> Alex
> >>>
> >>> On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gy...@gmail.com> wrote:
> >>>>
> >>>> Hey all!
> >>>>
> >>>> I would like to rekindle this discussion as it seems that it has
> >> stalled
> >>>> several times in the past and we are nearing the point in time where
> >> the
> >>>> decision has to be made with regards to 2.0. (we are already a bit
> late
> >>> but
> >>>> nevermind)
> >>>>
> >>>> There has been numerous requests and efforts to upgrade Kryo to better
> >>>> support newer Java versions and Java native types. I think we can all
> >>> agree
> >>>> that this change is inevitable one way or another.
> >>>>
> >>>> The latest JIRA for this seems to be:
> >>>> https://issues.apache.org/jira/browse/FLINK-3154
> >>>>
> >>>> There is even an open PR that accomplishes this (currently in a state
> >>>> incompatible way) but based on the discussion it seems that with some
> >>> extra
> >>>> complexity compatibility can even be preserved by having both the old
> >> and
> >>>> new Kryo versions active at the same time.
> >>>>
> >>>> The main question here is whether state compatibility is important for
> >>> 2.0
> >>>> with this regard or we want to bite the bullet and make this upgrade
> >> once
> >>>> and for all.
> >>>>
> >>>> Cheers,
> >>>> Gyula
> >>>
> >>
> >
>
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Timo Walther <tw...@apache.org>.
Hi Gyula,
thanks for bringing this up. I agree that we should use one of the most
recent Kryo versions for Flink 2.0. Otherwise the community will suffer
again from incompatibilities and needs to wait for another major Flink
version. Thanks for starting this discussion. We should do it for 2.0.
If not now, when?
Cheers,
Timo
On 24.02.25 08:03, Gyula Fóra wrote:
> Hey!
>
> I think the main question here is whether Flink 2.0 is going to be state
> backward compatible or not.
> If not then we need to make the upgrade right now, freeze or not. We have
> to decide this as a community.
>
> If we need to preserve backward compatibility then we need to go with a
> much more complex solution here when upgrading Kryo to do that.
>
> Do we have a community decision somewhere related to state
> backward compatibility for 2.0?
>
> Cheers
> Gyula
>
> On Mon, Feb 24, 2025 at 3:02 AM Rui Fan <19...@gmail.com> wrote:
>
>> Thanks Gyula for driving this discussion!
>>
>> +1 for upgrading kyro, I have a question about the timeline.
>> The flink 2.0.0 has been freezed, do we have enough time to
>> test if it's done in flink 2.0.0?
>>
>> Best,
>> Rui
>>
>> On Fri, Feb 21, 2025 at 9:55 PM Alexander Fedulov <af...@apache.org>
>> wrote:
>>
>>> Hi Gyula,
>>>
>>> Thanks for bringing up this topic! Kryo incompatibility with newer
>>> Java versions is a major issue that needs to be addressed, and in my
>>> opinion, Flink 2.0 provides a great opportunity to introduce this
>>> change.
>>>
>>> My understanding is that state compatibility was not a strict goal
>>> during 2.0 development. However, I’ve heard mentions that it might
>>> actually be compatible after all.
>>>
>>> From this perspective:
>>> definitely +1 on upgrading Kryo. The decision to invest additional
>>> effort in maintaining compatibility should depend on whether all other
>>> changes have preserved compatibility guarantees. If Kryo is the only
>>> breaking change, then ensuring compatibility might be worth
>>> considering.
>>>
>>> Best,
>>> Alex
>>>
>>> On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gy...@gmail.com> wrote:
>>>>
>>>> Hey all!
>>>>
>>>> I would like to rekindle this discussion as it seems that it has
>> stalled
>>>> several times in the past and we are nearing the point in time where
>> the
>>>> decision has to be made with regards to 2.0. (we are already a bit late
>>> but
>>>> nevermind)
>>>>
>>>> There has been numerous requests and efforts to upgrade Kryo to better
>>>> support newer Java versions and Java native types. I think we can all
>>> agree
>>>> that this change is inevitable one way or another.
>>>>
>>>> The latest JIRA for this seems to be:
>>>> https://issues.apache.org/jira/browse/FLINK-3154
>>>>
>>>> There is even an open PR that accomplishes this (currently in a state
>>>> incompatible way) but based on the discussion it seems that with some
>>> extra
>>>> complexity compatibility can even be preserved by having both the old
>> and
>>>> new Kryo versions active at the same time.
>>>>
>>>> The main question here is whether state compatibility is important for
>>> 2.0
>>>> with this regard or we want to bite the bullet and make this upgrade
>> once
>>>> and for all.
>>>>
>>>> Cheers,
>>>> Gyula
>>>
>>
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Gyula Fóra <gy...@gmail.com>.
Hey!
I think the main question here is whether Flink 2.0 is going to be state
backward compatible or not.
If not then we need to make the upgrade right now, freeze or not. We have
to decide this as a community.
If we need to preserve backward compatibility then we need to go with a
much more complex solution here when upgrading Kryo to do that.
Do we have a community decision somewhere related to state
backward compatibility for 2.0?
Cheers
Gyula
On Mon, Feb 24, 2025 at 3:02 AM Rui Fan <19...@gmail.com> wrote:
> Thanks Gyula for driving this discussion!
>
> +1 for upgrading kyro, I have a question about the timeline.
> The flink 2.0.0 has been freezed, do we have enough time to
> test if it's done in flink 2.0.0?
>
> Best,
> Rui
>
> On Fri, Feb 21, 2025 at 9:55 PM Alexander Fedulov <af...@apache.org>
> wrote:
>
> > Hi Gyula,
> >
> > Thanks for bringing up this topic! Kryo incompatibility with newer
> > Java versions is a major issue that needs to be addressed, and in my
> > opinion, Flink 2.0 provides a great opportunity to introduce this
> > change.
> >
> > My understanding is that state compatibility was not a strict goal
> > during 2.0 development. However, I’ve heard mentions that it might
> > actually be compatible after all.
> >
> > From this perspective:
> > definitely +1 on upgrading Kryo. The decision to invest additional
> > effort in maintaining compatibility should depend on whether all other
> > changes have preserved compatibility guarantees. If Kryo is the only
> > breaking change, then ensuring compatibility might be worth
> > considering.
> >
> > Best,
> > Alex
> >
> > On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gy...@gmail.com> wrote:
> > >
> > > Hey all!
> > >
> > > I would like to rekindle this discussion as it seems that it has
> stalled
> > > several times in the past and we are nearing the point in time where
> the
> > > decision has to be made with regards to 2.0. (we are already a bit late
> > but
> > > nevermind)
> > >
> > > There has been numerous requests and efforts to upgrade Kryo to better
> > > support newer Java versions and Java native types. I think we can all
> > agree
> > > that this change is inevitable one way or another.
> > >
> > > The latest JIRA for this seems to be:
> > > https://issues.apache.org/jira/browse/FLINK-3154
> > >
> > > There is even an open PR that accomplishes this (currently in a state
> > > incompatible way) but based on the discussion it seems that with some
> > extra
> > > complexity compatibility can even be preserved by having both the old
> and
> > > new Kryo versions active at the same time.
> > >
> > > The main question here is whether state compatibility is important for
> > 2.0
> > > with this regard or we want to bite the bullet and make this upgrade
> once
> > > and for all.
> > >
> > > Cheers,
> > > Gyula
> >
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Rui Fan <19...@gmail.com>.
Thanks Gyula for driving this discussion!
+1 for upgrading kyro, I have a question about the timeline.
The flink 2.0.0 has been freezed, do we have enough time to
test if it's done in flink 2.0.0?
Best,
Rui
On Fri, Feb 21, 2025 at 9:55 PM Alexander Fedulov <af...@apache.org>
wrote:
> Hi Gyula,
>
> Thanks for bringing up this topic! Kryo incompatibility with newer
> Java versions is a major issue that needs to be addressed, and in my
> opinion, Flink 2.0 provides a great opportunity to introduce this
> change.
>
> My understanding is that state compatibility was not a strict goal
> during 2.0 development. However, I’ve heard mentions that it might
> actually be compatible after all.
>
> From this perspective:
> definitely +1 on upgrading Kryo. The decision to invest additional
> effort in maintaining compatibility should depend on whether all other
> changes have preserved compatibility guarantees. If Kryo is the only
> breaking change, then ensuring compatibility might be worth
> considering.
>
> Best,
> Alex
>
> On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gy...@gmail.com> wrote:
> >
> > Hey all!
> >
> > I would like to rekindle this discussion as it seems that it has stalled
> > several times in the past and we are nearing the point in time where the
> > decision has to be made with regards to 2.0. (we are already a bit late
> but
> > nevermind)
> >
> > There has been numerous requests and efforts to upgrade Kryo to better
> > support newer Java versions and Java native types. I think we can all
> agree
> > that this change is inevitable one way or another.
> >
> > The latest JIRA for this seems to be:
> > https://issues.apache.org/jira/browse/FLINK-3154
> >
> > There is even an open PR that accomplishes this (currently in a state
> > incompatible way) but based on the discussion it seems that with some
> extra
> > complexity compatibility can even be preserved by having both the old and
> > new Kryo versions active at the same time.
> >
> > The main question here is whether state compatibility is important for
> 2.0
> > with this regard or we want to bite the bullet and make this upgrade once
> > and for all.
> >
> > Cheers,
> > Gyula
>
Re: [DISCUSSION] Upgrade to Kryo 5 for Flink 2.0
Posted by Alexander Fedulov <af...@apache.org>.
Hi Gyula,
Thanks for bringing up this topic! Kryo incompatibility with newer
Java versions is a major issue that needs to be addressed, and in my
opinion, Flink 2.0 provides a great opportunity to introduce this
change.
My understanding is that state compatibility was not a strict goal
during 2.0 development. However, I’ve heard mentions that it might
actually be compatible after all.
From this perspective:
definitely +1 on upgrading Kryo. The decision to invest additional
effort in maintaining compatibility should depend on whether all other
changes have preserved compatibility guarantees. If Kryo is the only
breaking change, then ensuring compatibility might be worth
considering.
Best,
Alex
On Fri, 21 Feb 2025 at 06:05, Gyula Fóra <gy...@gmail.com> wrote:
>
> Hey all!
>
> I would like to rekindle this discussion as it seems that it has stalled
> several times in the past and we are nearing the point in time where the
> decision has to be made with regards to 2.0. (we are already a bit late but
> nevermind)
>
> There has been numerous requests and efforts to upgrade Kryo to better
> support newer Java versions and Java native types. I think we can all agree
> that this change is inevitable one way or another.
>
> The latest JIRA for this seems to be:
> https://issues.apache.org/jira/browse/FLINK-3154
>
> There is even an open PR that accomplishes this (currently in a state
> incompatible way) but based on the discussion it seems that with some extra
> complexity compatibility can even be preserved by having both the old and
> new Kryo versions active at the same time.
>
> The main question here is whether state compatibility is important for 2.0
> with this regard or we want to bite the bullet and make this upgrade once
> and for all.
>
> Cheers,
> Gyula