You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Martijn Visser <ma...@apache.org> on 2023/03/30 12:25:51 UTC
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Hi all,
I also saw a thread on this topic from Clayton Wohl [1] on this topic,
which I'm including in this discussion thread to avoid that it gets lost.
From my perspective, there's two main ways to get to Java 17:
1. The Flink community agrees that we upgrade Kryo to a later version,
which means breaking all checkpoint/savepoint compatibility and releasing a
Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
dropped. This is probably the quickest way, but would still mean that we
expose Kryo in the Flink APIs, which is the main reason why we haven't been
able to upgrade Kryo at all.
2. There's a contributor who makes a contribution that bumps Kryo, but
either a) automagically reads in all old checkpoints/savepoints in using
Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
in the Kryo migration guide [2][3] or b) provides an offline tool that
allows users that are interested in migrating their snapshots manually
before starting from a newer version. That potentially could prevent the
need to introduce a new Flink major version. In both scenarios, ideally the
contributor would also help with avoiding the exposure of Kryo so that we
will be in a better shape in the future.
It would be good to get the opinion of the community for either of these
two options, or potentially for another one that I haven't mentioned. If it
appears that there's an overall agreement on the direction, I would propose
that a FLIP gets created which describes the entire process.
Looking forward to the thoughts of others, including the Users (therefore
including the User ML).
Best regards,
Martijn
[1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
[2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
[3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
wrote:
> I agree, there are several options to mitigate the migration from v2 to v5.
> yet, Oracle roadmap is to end JDK 11 support in September this year.
>
>
>
> ________________________________
> From: ConradJam <ja...@gmail.com>
> Sent: Thursday, March 16, 2023 4:36 AM
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
> (LTS)
>
> EXTERNAL EMAIL
>
>
>
> Thanks for your start this discuss
>
>
> I have been tracking this problem for a long time, until I saw a
> conversation in ISSUSE a few days ago and learned that the Kryo version
> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>
> As @cherry said it ruined our whole effort towards JDK17
>
> I am in favor of providing an external tool to migrate from Kryo old
> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
> tool start in flink 2.0 ?), does this tool currently have any plans or
> ideas worth discuss
>
>
> I think it should not be difficult to be compatible with JDK11 and JDK17.
> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
> that it is marked as Deprecated [2]
>
>
> Here I add that we need to pay attention to the version of Scala and the
> version of JDK17
>
>
> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> https://issues.apache.org/jira/browse/FLINK-24998
>
> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>
> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>
> > Hey dev community,
> >
> > I'm writing this email to kick off a discussion following this epic:
> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >
> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
> with
> > no timeline, the reason, based on the aforementioned ticket is the
> > following tickets
> >
> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> > https://issues.apache.org/jira/browse/FLINK-24998>.
> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >
> > My question is whether it is possible to release a major version (Flink
> > 2.0.0) using the latest Kryo version for those who don't need to restore
> > old savepoints/checkpoints in newer format.
> >
> > 1. Leverage JDK 17 features within JVM
> > 2. Moving from the old format to the newer one will be handled only
> > once - a mitigation can be achieved by a conversion tool or external
> > serializers, both can be provided later on.
> >
> > I'd like to emphasize that the next JDK LTS (21) will be released this
> > September. furthermore, Flink already supports JDK 12-15, which is very
> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11 will
> > become a legacy soon, as more frameworks moving towards JDK 17 and are
> less
> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
> > requires JDK 17 already).
> >
> > Thank you for your consideration of my request.
> >
> > Tamir.
> >
> >
> >
> >
> > Confidentiality: This communication and any attachments are intended for
> > the above-named persons only and may be confidential and/or legally
> > privileged. Any opinions expressed in this communication are not
> > necessarily those of NICE Actimize. If this communication has come to you
> > in error you must take no action based on it, nor must you copy or show
> it
> > to anyone; please delete/destroy and inform the sender by e-mail
> > immediately.
> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring that this e-mail
> and
> > attachments are free from any virus, we advise that in keeping with good
> > computing practice the recipient should ensure they are actually virus
> free.
> >
>
>
> --
> Best
>
> ConradJam
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Eric Brzezenski <er...@gmail.com>.
I realize I'm not looking at all angles here, but the biggest hitch that I
see with including both serializers and a migration process is that it
requires Flink and its users to stay on JDK < 17 for at least one "bump"
version before moving up. At this time they can upgrade to the version of
Flink without Kryo v2, built with JDK 17. It's a rather painful request to
ask customers to take on.
This feels like the process needs to live outside of Flink so a seamless
upgrade path can be supported for both customers and developers.
On Thu, Mar 30, 2023 at 8:27 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi all,
>
> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
> which I'm including in this discussion thread to avoid that it gets lost.
>
> From my perspective, there's two main ways to get to Java 17:
>
> 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
> 2. There's a contributor who makes a contribution that bumps Kryo, but
> either a) automagically reads in all old checkpoints/savepoints in using
> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
> in the Kryo migration guide [2][3] or b) provides an offline tool that
> allows users that are interested in migrating their snapshots manually
> before starting from a newer version. That potentially could prevent the
> need to introduce a new Flink major version. In both scenarios, ideally the
> contributor would also help with avoiding the exposure of Kryo so that we
> will be in a better shape in the future.
>
> It would be good to get the opinion of the community for either of these
> two options, or potentially for another one that I haven't mentioned. If it
> appears that there's an overall agreement on the direction, I would propose
> that a FLIP gets created which describes the entire process.
>
> Looking forward to the thoughts of others, including the Users (therefore
> including the User ML).
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>
> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
>
> > I agree, there are several options to mitigate the migration from v2 to
> v5.
> > yet, Oracle roadmap is to end JDK 11 support in September this year.
> >
> >
> >
> > ________________________________
> > From: ConradJam <ja...@gmail.com>
> > Sent: Thursday, March 16, 2023 4:36 AM
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [Discussion] - Release major Flink version to support JDK 17
> > (LTS)
> >
> > EXTERNAL EMAIL
> >
> >
> >
> > Thanks for your start this discuss
> >
> >
> > I have been tracking this problem for a long time, until I saw a
> > conversation in ISSUSE a few days ago and learned that the Kryo version
> > problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
> >
> > As @cherry said it ruined our whole effort towards JDK17
> >
> > I am in favor of providing an external tool to migrate from Kryo old
> > version checkpoint to the new Kryo new checkpoint at one time (Maybe this
> > tool start in flink 2.0 ?), does this tool currently have any plans or
> > ideas worth discuss
> >
> >
> > I think it should not be difficult to be compatible with JDK11 and JDK17.
> > We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
> > that it is marked as Deprecated [2]
> >
> >
> > Here I add that we need to pay attention to the version of Scala and the
> > version of JDK17
> >
> >
> > [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> > https://issues.apache.org/jira/browse/FLINK-24998
> >
> > [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
> instead
> > of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
> >
> > Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
> >
> > > Hey dev community,
> > >
> > > I'm writing this email to kick off a discussion following this epic:
> > > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> > >
> > > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
> which
> > > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
> > with
> > > no timeline, the reason, based on the aforementioned ticket is the
> > > following tickets
> > >
> > > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> > > https://issues.apache.org/jira/browse/FLINK-24998>.
> > > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> > > version<https://issues.apache.org/jira/browse/FLINK-3154>
> > >
> > > My question is whether it is possible to release a major version (Flink
> > > 2.0.0) using the latest Kryo version for those who don't need to
> restore
> > > old savepoints/checkpoints in newer format.
> > >
> > > 1. Leverage JDK 17 features within JVM
> > > 2. Moving from the old format to the newer one will be handled only
> > > once - a mitigation can be achieved by a conversion tool or external
> > > serializers, both can be provided later on.
> > >
> > > I'd like to emphasize that the next JDK LTS (21) will be released this
> > > September. furthermore, Flink already supports JDK 12-15, which is
> very
> > > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
> will
> > > become a legacy soon, as more frameworks moving towards JDK 17 and are
> > less
> > > likely to support JDK 11 in the near future. (For example, Spring Boot
> 3
> > > requires JDK 17 already).
> > >
> > > Thank you for your consideration of my request.
> > >
> > > Tamir.
> > >
> > >
> > >
> > >
> > > Confidentiality: This communication and any attachments are intended
> for
> > > the above-named persons only and may be confidential and/or legally
> > > privileged. Any opinions expressed in this communication are not
> > > necessarily those of NICE Actimize. If this communication has come to
> you
> > > in error you must take no action based on it, nor must you copy or show
> > it
> > > to anyone; please delete/destroy and inform the sender by e-mail
> > > immediately.
> > > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> > > Viruses: Although we have taken steps toward ensuring that this e-mail
> > and
> > > attachments are free from any virus, we advise that in keeping with
> good
> > > computing practice the recipient should ensure they are actually virus
> > free.
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
> > Confidentiality: This communication and any attachments are intended for
> > the above-named persons only and may be confidential and/or legally
> > privileged. Any opinions expressed in this communication are not
> > necessarily those of NICE Actimize. If this communication has come to you
> > in error you must take no action based on it, nor must you copy or show
> it
> > to anyone; please delete/destroy and inform the sender by e-mail
> > immediately.
> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring that this e-mail
> and
> > attachments are free from any virus, we advise that in keeping with good
> > computing practice the recipient should ensure they are actually virus
> free.
> >
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Tamir Sagi <Ta...@niceactimize.com>.
Hey Chesnay,
I'm sending a follow up email regarding JDK 17 support.
I see the Epic[1] is in progress and frequently updated. I'm curios if there is an ETA or any plan to release a major release with JDK 17 support that is not backward compatible.
Thanks,
Tamir
[1] https://issues.apache.org/jira/browse/FLINK-15736
________________________________
From: Alexey Novakov <al...@ververica.com>
Sent: Friday, April 28, 2023 11:41 AM
To: Chesnay Schepler <ch...@apache.org>
Cc: Thomas Weise <th...@apache.org>; dev@flink.apache.org <de...@flink.apache.org>; Jing Ge <ji...@ververica.com>; Tamir Sagi <Ta...@niceactimize.com>; Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <sa...@gmail.com>; user <us...@flink.apache.org>
Subject: Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
EXTERNAL EMAIL
If breaking savepoint compatibility will be eventually an option, I would recommend to try to upgrade Flink's Scala even to 2.13
Best regards,
Alexey
On Fri, Apr 28, 2023 at 10:22 AM Chesnay Schepler <ch...@apache.org>> wrote:
We don't know yet. I wanted to run some more experiments to see if I cant get Scala 2.12.7 working on Java 17.
If that doesn't work, then it would also be an option to bump Scala in the Java 17 builds (breaking savepoint compatibility), and users should just only use the Java APIs.
The alternative to _that_ is doing this when we drop the Scala API.
On 28/04/2023 01:11, Thomas Weise wrote:
Is the intention to bump the Flink major version and only support Java 17+? If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser <ma...@apache.org>> wrote:
Scala 2.12.7 doesn't compile on Java 17, see
https://issues.apache.org/jira/browse/FLINK-25000.
On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com>> wrote:
> Thanks Tamir for the information. According to the latest comment of the
> task FLINK-24998, this bug should be gone while using the latest JDK 17. I
> was wondering whether it means that there are no more issues to stop us
> releasing a major Flink version to support Java 17? Did I miss something?
>
> Best regards,
> Jing
>
> On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>>
> wrote:
>
>> More details about the JDK bug here
>> https://bugs.openjdk.org/browse/JDK-8277529
>>
>> Related Jira ticket
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> ------------------------------
>> *From:* Jing Ge via user <us...@flink.apache.org>>
>> *Sent:* Monday, April 24, 2023 11:15 PM
>> *To:* Chesnay Schepler <ch...@apache.org>>
>> *Cc:* Piotr Nowojski <pn...@apache.org>>; Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com<ma...@gmail.com>>; Martijn Visser <ma...@apache.org>>;
>> dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>; user <us...@flink.apache.org>>
>> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
>> 17 (LTS)
>>
>>
>> *EXTERNAL EMAIL*
>>
>>
>> Thanks Chesnay for working on this. Would you like to share more info
>> about the JDK bug?
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>>
>> wrote:
>>
>> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>>
>> On 31/03/2023 08:57, Chesnay Schepler wrote:
>>
>>
>> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>>
>> Kroy themselves state that v5 likely can't read v2 data.
>>
>> However, both versions can be on the classpath without classpath as v5
>> offers a versioned artifact that includes the version in the package.
>>
>> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
>> purely from a read/write perspective.
>>
>> The bigger question is how we expose this new Kryo version in the API. If
>> we stick to the versioned jar we need to either duplicate all current
>> Kryo-related APIs or find a better way to integrate other serialization
>> stacks.
>> On 30/03/2023 17:50, Piotr Nowojski wrote:
>>
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth migration
>> from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>>
>> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvisser@apache.org<ma...@apache.org>>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> which I'm including in this discussion thread to avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots manually
>> before starting from a newer version. That potentially could prevent the
>> need to introduce a new Flink major version. In both scenarios, ideally the
>> contributor would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either of these
>> two options, or potentially for another one that I haven't mentioned. If it
>> appears that there's an overall agreement on the direction, I would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users (therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>>
>> wrote:
>>
>> I agree, there are several options to mitigate the migration from v2 to
>> v5.
>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
>> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
>> (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until I saw a
>> conversation in ISSUSE a few days ago and learned that the Kryo version
>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com>> 于2023年3月16日周四 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion following this epic:
>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> with
>> > no timeline, the reason, based on the aforementioned ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a major version (Flink
>> > 2.0.0) using the latest Kryo version for those who don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will be handled only
>> > once - a mitigation can be achieved by a conversion tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>> > September. furthermore, Flink already supports JDK 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> will
>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>> less
>> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments are intended for
>> > the above-named persons only and may be confidential and/or legally
>> > privileged. Any opinions expressed in this communication are not
>> > necessarily those of NICE Actimize. If this communication has come to
>> you
>> > in error you must take no action based on it, nor must you copy or show
>> it
>> > to anyone; please delete/destroy and inform the sender by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>> and
>> > attachments are free from any virus, we advise that in keeping with good
>> > computing practice the recipient should ensure they are actually virus
>> free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>>
>>
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Tamir Sagi <Ta...@niceactimize.com>.
Hey Chesnay,
I'm sending a follow up email regarding JDK 17 support.
I see the Epic[1] is in progress and frequently updated. I'm curios if there is an ETA or any plan to release a major release with JDK 17 support that is not backward compatible.
Thanks,
Tamir
[1] https://issues.apache.org/jira/browse/FLINK-15736
________________________________
From: Alexey Novakov <al...@ververica.com>
Sent: Friday, April 28, 2023 11:41 AM
To: Chesnay Schepler <ch...@apache.org>
Cc: Thomas Weise <th...@apache.org>; dev@flink.apache.org <de...@flink.apache.org>; Jing Ge <ji...@ververica.com>; Tamir Sagi <Ta...@niceactimize.com>; Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <sa...@gmail.com>; user <us...@flink.apache.org>
Subject: Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
EXTERNAL EMAIL
If breaking savepoint compatibility will be eventually an option, I would recommend to try to upgrade Flink's Scala even to 2.13
Best regards,
Alexey
On Fri, Apr 28, 2023 at 10:22 AM Chesnay Schepler <ch...@apache.org>> wrote:
We don't know yet. I wanted to run some more experiments to see if I cant get Scala 2.12.7 working on Java 17.
If that doesn't work, then it would also be an option to bump Scala in the Java 17 builds (breaking savepoint compatibility), and users should just only use the Java APIs.
The alternative to _that_ is doing this when we drop the Scala API.
On 28/04/2023 01:11, Thomas Weise wrote:
Is the intention to bump the Flink major version and only support Java 17+? If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser <ma...@apache.org>> wrote:
Scala 2.12.7 doesn't compile on Java 17, see
https://issues.apache.org/jira/browse/FLINK-25000.
On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com>> wrote:
> Thanks Tamir for the information. According to the latest comment of the
> task FLINK-24998, this bug should be gone while using the latest JDK 17. I
> was wondering whether it means that there are no more issues to stop us
> releasing a major Flink version to support Java 17? Did I miss something?
>
> Best regards,
> Jing
>
> On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>>
> wrote:
>
>> More details about the JDK bug here
>> https://bugs.openjdk.org/browse/JDK-8277529
>>
>> Related Jira ticket
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> ------------------------------
>> *From:* Jing Ge via user <us...@flink.apache.org>>
>> *Sent:* Monday, April 24, 2023 11:15 PM
>> *To:* Chesnay Schepler <ch...@apache.org>>
>> *Cc:* Piotr Nowojski <pn...@apache.org>>; Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com<ma...@gmail.com>>; Martijn Visser <ma...@apache.org>>;
>> dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>; user <us...@flink.apache.org>>
>> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
>> 17 (LTS)
>>
>>
>> *EXTERNAL EMAIL*
>>
>>
>> Thanks Chesnay for working on this. Would you like to share more info
>> about the JDK bug?
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>>
>> wrote:
>>
>> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>>
>> On 31/03/2023 08:57, Chesnay Schepler wrote:
>>
>>
>> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>>
>> Kroy themselves state that v5 likely can't read v2 data.
>>
>> However, both versions can be on the classpath without classpath as v5
>> offers a versioned artifact that includes the version in the package.
>>
>> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
>> purely from a read/write perspective.
>>
>> The bigger question is how we expose this new Kryo version in the API. If
>> we stick to the versioned jar we need to either duplicate all current
>> Kryo-related APIs or find a better way to integrate other serialization
>> stacks.
>> On 30/03/2023 17:50, Piotr Nowojski wrote:
>>
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth migration
>> from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>>
>> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvisser@apache.org<ma...@apache.org>>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> which I'm including in this discussion thread to avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots manually
>> before starting from a newer version. That potentially could prevent the
>> need to introduce a new Flink major version. In both scenarios, ideally the
>> contributor would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either of these
>> two options, or potentially for another one that I haven't mentioned. If it
>> appears that there's an overall agreement on the direction, I would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users (therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>>
>> wrote:
>>
>> I agree, there are several options to mitigate the migration from v2 to
>> v5.
>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
>> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
>> (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until I saw a
>> conversation in ISSUSE a few days ago and learned that the Kryo version
>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com>> 于2023年3月16日周四 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion following this epic:
>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> with
>> > no timeline, the reason, based on the aforementioned ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a major version (Flink
>> > 2.0.0) using the latest Kryo version for those who don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will be handled only
>> > once - a mitigation can be achieved by a conversion tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>> > September. furthermore, Flink already supports JDK 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> will
>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>> less
>> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments are intended for
>> > the above-named persons only and may be confidential and/or legally
>> > privileged. Any opinions expressed in this communication are not
>> > necessarily those of NICE Actimize. If this communication has come to
>> you
>> > in error you must take no action based on it, nor must you copy or show
>> it
>> > to anyone; please delete/destroy and inform the sender by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>> and
>> > attachments are free from any virus, we advise that in keeping with good
>> > computing practice the recipient should ensure they are actually virus
>> free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>>
>>
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Alexey Novakov via user <us...@flink.apache.org>.
If breaking savepoint compatibility will be eventually an option, I would
recommend to try to upgrade Flink's Scala even to *2.13*
Best regards,
Alexey
On Fri, Apr 28, 2023 at 10:22 AM Chesnay Schepler <ch...@apache.org>
wrote:
> We don't know yet. I wanted to run some more experiments to see if I cant
> get Scala 2.12.7 working on Java 17.
>
> If that doesn't work, then it would also be an option to bump Scala in the
> Java 17 builds (breaking savepoint compatibility), and users should just
> only use the Java APIs.
>
> The alternative to _that_ is doing this when we drop the Scala API.
>
> On 28/04/2023 01:11, Thomas Weise wrote:
>
> Is the intention to bump the Flink major version and only support Java
> 17+? If so, can Scala not be upgraded at the same time?
>
> Thanks,
> Thomas
>
>
> On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Scala 2.12.7 doesn't compile on Java 17, see
>> https://issues.apache.org/jira/browse/FLINK-25000.
>>
>> On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>>
>> > Thanks Tamir for the information. According to the latest comment of the
>> > task FLINK-24998, this bug should be gone while using the latest JDK
>> 17. I
>> > was wondering whether it means that there are no more issues to stop us
>> > releasing a major Flink version to support Java 17? Did I miss
>> something?
>> >
>> > Best regards,
>> > Jing
>> >
>> > On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Tamir.Sagi@niceactimize.com
>> >
>> > wrote:
>> >
>> >> More details about the JDK bug here
>> >> https://bugs.openjdk.org/browse/JDK-8277529
>> >>
>> >> Related Jira ticket
>> >> https://issues.apache.org/jira/browse/FLINK-24998
>> >>
>> >> ------------------------------
>> >> *From:* Jing Ge via user <us...@flink.apache.org>
>> >> *Sent:* Monday, April 24, 2023 11:15 PM
>> >> *To:* Chesnay Schepler <ch...@apache.org>
>> >> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
>> >> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
>> >> dev@flink.apache.org <de...@flink.apache.org>; user <
>> user@flink.apache.org>
>> >> *Subject:* Re: [Discussion] - Release major Flink version to support
>> JDK
>> >> 17 (LTS)
>> >>
>> >>
>> >> *EXTERNAL EMAIL*
>> >>
>> >>
>> >> Thanks Chesnay for working on this. Would you like to share more info
>> >> about the JDK bug?
>> >>
>> >> Best regards,
>> >> Jing
>> >>
>> >> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
>> >> wrote:
>> >>
>> >> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>> >>
>> >> On 31/03/2023 08:57, Chesnay Schepler wrote:
>> >>
>> >>
>> >>
>> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>> >>
>> >> Kroy themselves state that v5 likely can't read v2 data.
>> >>
>> >> However, both versions can be on the classpath without classpath as v5
>> >> offers a versioned artifact that includes the version in the package.
>> >>
>> >> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
>> >> purely from a read/write perspective.
>> >>
>> >> The bigger question is how we expose this new Kryo version in the API.
>> If
>> >> we stick to the versioned jar we need to either duplicate all current
>> >> Kryo-related APIs or find a better way to integrate other serialization
>> >> stacks.
>> >> On 30/03/2023 17:50, Piotr Nowojski wrote:
>> >>
>> >> Hey,
>> >>
>> >> > 1. The Flink community agrees that we upgrade Kryo to a later
>> version,
>> >> which means breaking all checkpoint/savepoint compatibility and
>> releasing a
>> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
>> support
>> >> dropped. This is probably the quickest way, but would still mean that
>> we
>> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
>> been
>> >> able to upgrade Kryo at all.
>> >>
>> >> This sounds pretty bad to me.
>> >>
>> >> Has anyone looked into what it would take to provide a smooth migration
>> >> from Kryo2 -> Kryo5?
>> >>
>> >> Best,
>> >> Piotrek
>> >>
>> >> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com>
>> >> napisał(a):
>> >>
>> >> Hi Martijn,
>> >>
>> >> just to be sure, if all state-related classes use a POJO serializer,
>> Kryo
>> >> will never come into play, right? Given FLINK-16686 [1], I wonder how
>> many
>> >> users actually have jobs with Kryo and RocksDB, but even if there
>> aren't
>> >> many, that still leaves those who don't use RocksDB for
>> >> checkpoints/savepoints.
>> >>
>> >> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> >> users choose between v2/v5 jars by separating them like log4j2 jars?
>> >>
>> >> [1] https://issues.apache.org/jira/browse/FLINK-16686
>> >>
>> >> Regards,
>> >> Alexis.
>> >>
>> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> >> martijnvisser@apache.org>:
>> >>
>> >> Hi all,
>> >>
>> >> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> >> which I'm including in this discussion thread to avoid that it gets
>> lost.
>> >>
>> >> From my perspective, there's two main ways to get to Java 17:
>> >>
>> >> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> >> which means breaking all checkpoint/savepoint compatibility and
>> releasing a
>> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
>> support
>> >> dropped. This is probably the quickest way, but would still mean that
>> we
>> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
>> been
>> >> able to upgrade Kryo at all.
>> >> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> >> either a) automagically reads in all old checkpoints/savepoints in
>> using
>> >> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
>> mentioned
>> >> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> >> allows users that are interested in migrating their snapshots manually
>> >> before starting from a newer version. That potentially could prevent
>> the
>> >> need to introduce a new Flink major version. In both scenarios,
>> ideally the
>> >> contributor would also help with avoiding the exposure of Kryo so that
>> we
>> >> will be in a better shape in the future.
>> >>
>> >> It would be good to get the opinion of the community for either of
>> these
>> >> two options, or potentially for another one that I haven't mentioned.
>> If it
>> >> appears that there's an overall agreement on the direction, I would
>> propose
>> >> that a FLIP gets created which describes the entire process.
>> >>
>> >> Looking forward to the thoughts of others, including the Users
>> (therefore
>> >> including the User ML).
>> >>
>> >> Best regards,
>> >>
>> >> Martijn
>> >>
>> >> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> >> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> >> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>> >>
>> >> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <
>> Tamir.Sagi@niceactimize.com>
>> >> wrote:
>> >>
>> >> I agree, there are several options to mitigate the migration from v2 to
>> >> v5.
>> >> yet, Oracle roadmap is to end JDK 11 support in September this year.
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> From: ConradJam <ja...@gmail.com>
>> >> Sent: Thursday, March 16, 2023 4:36 AM
>> >> To: dev@flink.apache.org <de...@flink.apache.org>
>> >> Subject: Re: [Discussion] - Release major Flink version to support JDK
>> 17
>> >> (LTS)
>> >>
>> >> EXTERNAL EMAIL
>> >>
>> >>
>> >>
>> >> Thanks for your start this discuss
>> >>
>> >>
>> >> I have been tracking this problem for a long time, until I saw a
>> >> conversation in ISSUSE a few days ago and learned that the Kryo version
>> >> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998
>> ,
>> >>
>> >> As @cherry said it ruined our whole effort towards JDK17
>> >>
>> >> I am in favor of providing an external tool to migrate from Kryo old
>> >> version checkpoint to the new Kryo new checkpoint at one time (Maybe
>> this
>> >> tool start in flink 2.0 ?), does this tool currently have any plans or
>> >> ideas worth discuss
>> >>
>> >>
>> >> I think it should not be difficult to be compatible with JDK11 and
>> JDK17.
>> >> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> >> that it is marked as Deprecated [2]
>> >>
>> >>
>> >> Here I add that we need to pay attention to the version of Scala and
>> the
>> >> version of JDK17
>> >>
>> >>
>> >> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> >> https://issues.apache.org/jira/browse/FLINK-24998
>> >>
>> >> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
>> instead
>> >> of requiring Java 11
>> https://issues.apache.org/jira/browse/FLINK-30501
>> >>
>> >> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>> >>
>> >> > Hey dev community,
>> >> >
>> >> > I'm writing this email to kick off a discussion following this epic:
>> >> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >> >
>> >> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
>> which
>> >> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> >> with
>> >> > no timeline, the reason, based on the aforementioned ticket is the
>> >> > following tickets
>> >> >
>> >> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> >> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> >> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> >> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >> >
>> >> > My question is whether it is possible to release a major version
>> (Flink
>> >> > 2.0.0) using the latest Kryo version for those who don't need to
>> restore
>> >> > old savepoints/checkpoints in newer format.
>> >> >
>> >> > 1. Leverage JDK 17 features within JVM
>> >> > 2. Moving from the old format to the newer one will be handled
>> only
>> >> > once - a mitigation can be achieved by a conversion tool or external
>> >> > serializers, both can be provided later on.
>> >> >
>> >> > I'd like to emphasize that the next JDK LTS (21) will be released
>> this
>> >> > September. furthermore, Flink already supports JDK 12-15, which is
>> very
>> >> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> >> will
>> >> > become a legacy soon, as more frameworks moving towards JDK 17 and
>> are
>> >> less
>> >> > likely to support JDK 11 in the near future. (For example, Spring
>> Boot 3
>> >> > requires JDK 17 already).
>> >> >
>> >> > Thank you for your consideration of my request.
>> >> >
>> >> > Tamir.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Confidentiality: This communication and any attachments are intended
>> for
>> >> > the above-named persons only and may be confidential and/or legally
>> >> > privileged. Any opinions expressed in this communication are not
>> >> > necessarily those of NICE Actimize. If this communication has come to
>> >> you
>> >> > in error you must take no action based on it, nor must you copy or
>> show
>> >> it
>> >> > to anyone; please delete/destroy and inform the sender by e-mail
>> >> > immediately.
>> >> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> >> > Viruses: Although we have taken steps toward ensuring that this
>> e-mail
>> >> and
>> >> > attachments are free from any virus, we advise that in keeping with
>> good
>> >> > computing practice the recipient should ensure they are actually
>> virus
>> >> free.
>> >> >
>> >>
>> >>
>> >> --
>> >> Best
>> >>
>> >> ConradJam
>> >>
>> >> Confidentiality: This communication and any attachments are intended
>> for
>> >> the above-named persons only and may be confidential and/or legally
>> >> privileged. Any opinions expressed in this communication are not
>> >> necessarily those of NICE Actimize. If this communication has come to
>> you
>> >> in error you must take no action based on it, nor must you copy or
>> show it
>> >> to anyone; please delete/destroy and inform the sender by e-mail
>> >> immediately.
>> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> >> Viruses: Although we have taken steps toward ensuring that this e-mail
>> >> and attachments are free from any virus, we advise that in keeping with
>> >> good computing practice the recipient should ensure they are actually
>> virus
>> >> free.
>> >>
>> >>
>> >>
>> >>
>> >> Confidentiality: This communication and any attachments are intended
>> for
>> >> the above-named persons only and may be confidential and/or legally
>> >> privileged. Any opinions expressed in this communication are not
>> >> necessarily those of NICE Actimize. If this communication has come to
>> you
>> >> in error you must take no action based on it, nor must you copy or
>> show it
>> >> to anyone; please delete/destroy and inform the sender by e-mail
>> >> immediately.
>> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> >> Viruses: Although we have taken steps toward ensuring that this e-mail
>> >> and attachments are free from any virus, we advise that in keeping with
>> >> good computing practice the recipient should ensure they are actually
>> virus
>> >> free.
>> >>
>> >
>>
>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Alexey Novakov <al...@ververica.com.INVALID>.
If breaking savepoint compatibility will be eventually an option, I would
recommend to try to upgrade Flink's Scala even to *2.13*
Best regards,
Alexey
On Fri, Apr 28, 2023 at 10:22 AM Chesnay Schepler <ch...@apache.org>
wrote:
> We don't know yet. I wanted to run some more experiments to see if I cant
> get Scala 2.12.7 working on Java 17.
>
> If that doesn't work, then it would also be an option to bump Scala in the
> Java 17 builds (breaking savepoint compatibility), and users should just
> only use the Java APIs.
>
> The alternative to _that_ is doing this when we drop the Scala API.
>
> On 28/04/2023 01:11, Thomas Weise wrote:
>
> Is the intention to bump the Flink major version and only support Java
> 17+? If so, can Scala not be upgraded at the same time?
>
> Thanks,
> Thomas
>
>
> On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Scala 2.12.7 doesn't compile on Java 17, see
>> https://issues.apache.org/jira/browse/FLINK-25000.
>>
>> On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>>
>> > Thanks Tamir for the information. According to the latest comment of the
>> > task FLINK-24998, this bug should be gone while using the latest JDK
>> 17. I
>> > was wondering whether it means that there are no more issues to stop us
>> > releasing a major Flink version to support Java 17? Did I miss
>> something?
>> >
>> > Best regards,
>> > Jing
>> >
>> > On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Tamir.Sagi@niceactimize.com
>> >
>> > wrote:
>> >
>> >> More details about the JDK bug here
>> >> https://bugs.openjdk.org/browse/JDK-8277529
>> >>
>> >> Related Jira ticket
>> >> https://issues.apache.org/jira/browse/FLINK-24998
>> >>
>> >> ------------------------------
>> >> *From:* Jing Ge via user <us...@flink.apache.org>
>> >> *Sent:* Monday, April 24, 2023 11:15 PM
>> >> *To:* Chesnay Schepler <ch...@apache.org>
>> >> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
>> >> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
>> >> dev@flink.apache.org <de...@flink.apache.org>; user <
>> user@flink.apache.org>
>> >> *Subject:* Re: [Discussion] - Release major Flink version to support
>> JDK
>> >> 17 (LTS)
>> >>
>> >>
>> >> *EXTERNAL EMAIL*
>> >>
>> >>
>> >> Thanks Chesnay for working on this. Would you like to share more info
>> >> about the JDK bug?
>> >>
>> >> Best regards,
>> >> Jing
>> >>
>> >> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
>> >> wrote:
>> >>
>> >> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>> >>
>> >> On 31/03/2023 08:57, Chesnay Schepler wrote:
>> >>
>> >>
>> >>
>> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>> >>
>> >> Kroy themselves state that v5 likely can't read v2 data.
>> >>
>> >> However, both versions can be on the classpath without classpath as v5
>> >> offers a versioned artifact that includes the version in the package.
>> >>
>> >> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
>> >> purely from a read/write perspective.
>> >>
>> >> The bigger question is how we expose this new Kryo version in the API.
>> If
>> >> we stick to the versioned jar we need to either duplicate all current
>> >> Kryo-related APIs or find a better way to integrate other serialization
>> >> stacks.
>> >> On 30/03/2023 17:50, Piotr Nowojski wrote:
>> >>
>> >> Hey,
>> >>
>> >> > 1. The Flink community agrees that we upgrade Kryo to a later
>> version,
>> >> which means breaking all checkpoint/savepoint compatibility and
>> releasing a
>> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
>> support
>> >> dropped. This is probably the quickest way, but would still mean that
>> we
>> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
>> been
>> >> able to upgrade Kryo at all.
>> >>
>> >> This sounds pretty bad to me.
>> >>
>> >> Has anyone looked into what it would take to provide a smooth migration
>> >> from Kryo2 -> Kryo5?
>> >>
>> >> Best,
>> >> Piotrek
>> >>
>> >> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com>
>> >> napisał(a):
>> >>
>> >> Hi Martijn,
>> >>
>> >> just to be sure, if all state-related classes use a POJO serializer,
>> Kryo
>> >> will never come into play, right? Given FLINK-16686 [1], I wonder how
>> many
>> >> users actually have jobs with Kryo and RocksDB, but even if there
>> aren't
>> >> many, that still leaves those who don't use RocksDB for
>> >> checkpoints/savepoints.
>> >>
>> >> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> >> users choose between v2/v5 jars by separating them like log4j2 jars?
>> >>
>> >> [1] https://issues.apache.org/jira/browse/FLINK-16686
>> >>
>> >> Regards,
>> >> Alexis.
>> >>
>> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> >> martijnvisser@apache.org>:
>> >>
>> >> Hi all,
>> >>
>> >> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> >> which I'm including in this discussion thread to avoid that it gets
>> lost.
>> >>
>> >> From my perspective, there's two main ways to get to Java 17:
>> >>
>> >> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> >> which means breaking all checkpoint/savepoint compatibility and
>> releasing a
>> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
>> support
>> >> dropped. This is probably the quickest way, but would still mean that
>> we
>> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
>> been
>> >> able to upgrade Kryo at all.
>> >> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> >> either a) automagically reads in all old checkpoints/savepoints in
>> using
>> >> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
>> mentioned
>> >> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> >> allows users that are interested in migrating their snapshots manually
>> >> before starting from a newer version. That potentially could prevent
>> the
>> >> need to introduce a new Flink major version. In both scenarios,
>> ideally the
>> >> contributor would also help with avoiding the exposure of Kryo so that
>> we
>> >> will be in a better shape in the future.
>> >>
>> >> It would be good to get the opinion of the community for either of
>> these
>> >> two options, or potentially for another one that I haven't mentioned.
>> If it
>> >> appears that there's an overall agreement on the direction, I would
>> propose
>> >> that a FLIP gets created which describes the entire process.
>> >>
>> >> Looking forward to the thoughts of others, including the Users
>> (therefore
>> >> including the User ML).
>> >>
>> >> Best regards,
>> >>
>> >> Martijn
>> >>
>> >> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> >> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> >> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>> >>
>> >> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <
>> Tamir.Sagi@niceactimize.com>
>> >> wrote:
>> >>
>> >> I agree, there are several options to mitigate the migration from v2 to
>> >> v5.
>> >> yet, Oracle roadmap is to end JDK 11 support in September this year.
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> From: ConradJam <ja...@gmail.com>
>> >> Sent: Thursday, March 16, 2023 4:36 AM
>> >> To: dev@flink.apache.org <de...@flink.apache.org>
>> >> Subject: Re: [Discussion] - Release major Flink version to support JDK
>> 17
>> >> (LTS)
>> >>
>> >> EXTERNAL EMAIL
>> >>
>> >>
>> >>
>> >> Thanks for your start this discuss
>> >>
>> >>
>> >> I have been tracking this problem for a long time, until I saw a
>> >> conversation in ISSUSE a few days ago and learned that the Kryo version
>> >> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998
>> ,
>> >>
>> >> As @cherry said it ruined our whole effort towards JDK17
>> >>
>> >> I am in favor of providing an external tool to migrate from Kryo old
>> >> version checkpoint to the new Kryo new checkpoint at one time (Maybe
>> this
>> >> tool start in flink 2.0 ?), does this tool currently have any plans or
>> >> ideas worth discuss
>> >>
>> >>
>> >> I think it should not be difficult to be compatible with JDK11 and
>> JDK17.
>> >> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> >> that it is marked as Deprecated [2]
>> >>
>> >>
>> >> Here I add that we need to pay attention to the version of Scala and
>> the
>> >> version of JDK17
>> >>
>> >>
>> >> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> >> https://issues.apache.org/jira/browse/FLINK-24998
>> >>
>> >> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
>> instead
>> >> of requiring Java 11
>> https://issues.apache.org/jira/browse/FLINK-30501
>> >>
>> >> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>> >>
>> >> > Hey dev community,
>> >> >
>> >> > I'm writing this email to kick off a discussion following this epic:
>> >> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >> >
>> >> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
>> which
>> >> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> >> with
>> >> > no timeline, the reason, based on the aforementioned ticket is the
>> >> > following tickets
>> >> >
>> >> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> >> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> >> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> >> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >> >
>> >> > My question is whether it is possible to release a major version
>> (Flink
>> >> > 2.0.0) using the latest Kryo version for those who don't need to
>> restore
>> >> > old savepoints/checkpoints in newer format.
>> >> >
>> >> > 1. Leverage JDK 17 features within JVM
>> >> > 2. Moving from the old format to the newer one will be handled
>> only
>> >> > once - a mitigation can be achieved by a conversion tool or external
>> >> > serializers, both can be provided later on.
>> >> >
>> >> > I'd like to emphasize that the next JDK LTS (21) will be released
>> this
>> >> > September. furthermore, Flink already supports JDK 12-15, which is
>> very
>> >> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> >> will
>> >> > become a legacy soon, as more frameworks moving towards JDK 17 and
>> are
>> >> less
>> >> > likely to support JDK 11 in the near future. (For example, Spring
>> Boot 3
>> >> > requires JDK 17 already).
>> >> >
>> >> > Thank you for your consideration of my request.
>> >> >
>> >> > Tamir.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Confidentiality: This communication and any attachments are intended
>> for
>> >> > the above-named persons only and may be confidential and/or legally
>> >> > privileged. Any opinions expressed in this communication are not
>> >> > necessarily those of NICE Actimize. If this communication has come to
>> >> you
>> >> > in error you must take no action based on it, nor must you copy or
>> show
>> >> it
>> >> > to anyone; please delete/destroy and inform the sender by e-mail
>> >> > immediately.
>> >> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> >> > Viruses: Although we have taken steps toward ensuring that this
>> e-mail
>> >> and
>> >> > attachments are free from any virus, we advise that in keeping with
>> good
>> >> > computing practice the recipient should ensure they are actually
>> virus
>> >> free.
>> >> >
>> >>
>> >>
>> >> --
>> >> Best
>> >>
>> >> ConradJam
>> >>
>> >> Confidentiality: This communication and any attachments are intended
>> for
>> >> the above-named persons only and may be confidential and/or legally
>> >> privileged. Any opinions expressed in this communication are not
>> >> necessarily those of NICE Actimize. If this communication has come to
>> you
>> >> in error you must take no action based on it, nor must you copy or
>> show it
>> >> to anyone; please delete/destroy and inform the sender by e-mail
>> >> immediately.
>> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> >> Viruses: Although we have taken steps toward ensuring that this e-mail
>> >> and attachments are free from any virus, we advise that in keeping with
>> >> good computing practice the recipient should ensure they are actually
>> virus
>> >> free.
>> >>
>> >>
>> >>
>> >>
>> >> Confidentiality: This communication and any attachments are intended
>> for
>> >> the above-named persons only and may be confidential and/or legally
>> >> privileged. Any opinions expressed in this communication are not
>> >> necessarily those of NICE Actimize. If this communication has come to
>> you
>> >> in error you must take no action based on it, nor must you copy or
>> show it
>> >> to anyone; please delete/destroy and inform the sender by e-mail
>> >> immediately.
>> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> >> Viruses: Although we have taken steps toward ensuring that this e-mail
>> >> and attachments are free from any virus, we advise that in keeping with
>> >> good computing practice the recipient should ensure they are actually
>> virus
>> >> free.
>> >>
>> >
>>
>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Chesnay Schepler <ch...@apache.org>.
We don't know yet. I wanted to run some more experiments to see if I
cant get Scala 2.12.7 working on Java 17.
If that doesn't work, then it would also be an option to bump Scala in
the Java 17 builds (breaking savepoint compatibility), and users should
just only use the Java APIs.
The alternative to _that_ is doing this when we drop the Scala API.
On 28/04/2023 01:11, Thomas Weise wrote:
> Is the intention to bump the Flink major version and only support Java
> 17+? If so, can Scala not be upgraded at the same time?
>
> Thanks,
> Thomas
>
>
> On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser
> <ma...@apache.org> wrote:
>
> Scala 2.12.7 doesn't compile on Java 17, see
> https://issues.apache.org/jira/browse/FLINK-25000.
>
> On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>
> > Thanks Tamir for the information. According to the latest
> comment of the
> > task FLINK-24998, this bug should be gone while using the latest
> JDK 17. I
> > was wondering whether it means that there are no more issues to
> stop us
> > releasing a major Flink version to support Java 17? Did I miss
> something?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi
> <Ta...@niceactimize.com>
> > wrote:
> >
> >> More details about the JDK bug here
> >> https://bugs.openjdk.org/browse/JDK-8277529
> >>
> >> Related Jira ticket
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> ------------------------------
> >> *From:* Jing Ge via user <us...@flink.apache.org>
> >> *Sent:* Monday, April 24, 2023 11:15 PM
> >> *To:* Chesnay Schepler <ch...@apache.org>
> >> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis
> Sarda-Espinosa <
> >> sarda.espinosa@gmail.com>; Martijn Visser
> <ma...@apache.org>;
> >> dev@flink.apache.org <de...@flink.apache.org>; user
> <us...@flink.apache.org>
> >> *Subject:* Re: [Discussion] - Release major Flink version to
> support JDK
> >> 17 (LTS)
> >>
> >>
> >> *EXTERNAL EMAIL*
> >>
> >>
> >> Thanks Chesnay for working on this. Would you like to share
> more info
> >> about the JDK bug?
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler
> <ch...@apache.org>
> >> wrote:
> >>
> >> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
> >>
> >> On 31/03/2023 08:57, Chesnay Schepler wrote:
> >>
> >>
> >>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> >>
> >> Kroy themselves state that v5 likely can't read v2 data.
> >>
> >> However, both versions can be on the classpath without
> classpath as v5
> >> offers a versioned artifact that includes the version in the
> package.
> >>
> >> It probably wouldn't be difficult to migrate a savepoint to
> Kryo v5,
> >> purely from a read/write perspective.
> >>
> >> The bigger question is how we expose this new Kryo version in
> the API. If
> >> we stick to the versioned jar we need to either duplicate all
> current
> >> Kryo-related APIs or find a better way to integrate other
> serialization
> >> stacks.
> >> On 30/03/2023 17:50, Piotr Nowojski wrote:
> >>
> >> Hey,
> >>
> >> > 1. The Flink community agrees that we upgrade Kryo to a later
> version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala
> API support
> >> dropped. This is probably the quickest way, but would still
> mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we
> haven't been
> >> able to upgrade Kryo at all.
> >>
> >> This sounds pretty bad to me.
> >>
> >> Has anyone looked into what it would take to provide a smooth
> migration
> >> from Kryo2 -> Kryo5?
> >>
> >> Best,
> >> Piotrek
> >>
> >> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
> <sa...@gmail.com>
> >> napisał(a):
> >>
> >> Hi Martijn,
> >>
> >> just to be sure, if all state-related classes use a POJO
> serializer, Kryo
> >> will never come into play, right? Given FLINK-16686 [1], I
> wonder how many
> >> users actually have jobs with Kryo and RocksDB, but even if
> there aren't
> >> many, that still leaves those who don't use RocksDB for
> >> checkpoints/savepoints.
> >>
> >> If Kryo were to stay in the Flink APIs in v1.X, is it
> impossible to let
> >> users choose between v2/v5 jars by separating them like log4j2
> jars?
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-16686
> >>
> >> Regards,
> >> Alexis.
> >>
> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> >> martijnvisser@apache.org>:
> >>
> >> Hi all,
> >>
> >> I also saw a thread on this topic from Clayton Wohl [1] on this
> topic,
> >> which I'm including in this discussion thread to avoid that it
> gets lost.
> >>
> >> From my perspective, there's two main ways to get to Java 17:
> >>
> >> 1. The Flink community agrees that we upgrade Kryo to a later
> version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala
> API support
> >> dropped. This is probably the quickest way, but would still
> mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we
> haven't been
> >> able to upgrade Kryo at all.
> >> 2. There's a contributor who makes a contribution that bumps
> Kryo, but
> >> either a) automagically reads in all old checkpoints/savepoints
> in using
> >> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
> mentioned
> >> in the Kryo migration guide [2][3] or b) provides an offline
> tool that
> >> allows users that are interested in migrating their snapshots
> manually
> >> before starting from a newer version. That potentially could
> prevent the
> >> need to introduce a new Flink major version. In both scenarios,
> ideally the
> >> contributor would also help with avoiding the exposure of Kryo
> so that we
> >> will be in a better shape in the future.
> >>
> >> It would be good to get the opinion of the community for either
> of these
> >> two options, or potentially for another one that I haven't
> mentioned. If it
> >> appears that there's an overall agreement on the direction, I
> would propose
> >> that a FLIP gets created which describes the entire process.
> >>
> >> Looking forward to the thoughts of others, including the Users
> (therefore
> >> including the User ML).
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> [1]
> https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> >> [2]
> https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> >> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
> >>
> >> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
> <Ta...@niceactimize.com>
> >> wrote:
> >>
> >> I agree, there are several options to mitigate the migration
> from v2 to
> >> v5.
> >> yet, Oracle roadmap is to end JDK 11 support in September this
> year.
> >>
> >>
> >>
> >> ________________________________
> >> From: ConradJam <ja...@gmail.com>
> >> Sent: Thursday, March 16, 2023 4:36 AM
> >> To: dev@flink.apache.org <de...@flink.apache.org>
> >> Subject: Re: [Discussion] - Release major Flink version to
> support JDK 17
> >> (LTS)
> >>
> >> EXTERNAL EMAIL
> >>
> >>
> >>
> >> Thanks for your start this discuss
> >>
> >>
> >> I have been tracking this problem for a long time, until I saw a
> >> conversation in ISSUSE a few days ago and learned that the Kryo
> version
> >> problem will affect the JDK17 compilation of snapshots [1]
> FLINK-24998 ,
> >>
> >> As @cherry said it ruined our whole effort towards JDK17
> >>
> >> I am in favor of providing an external tool to migrate from
> Kryo old
> >> version checkpoint to the new Kryo new checkpoint at one time
> (Maybe this
> >> tool start in flink 2.0 ?), does this tool currently have any
> plans or
> >> ideas worth discuss
> >>
> >>
> >> I think it should not be difficult to be compatible with JDK11
> and JDK17.
> >> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in
> the doc
> >> that it is marked as Deprecated [2]
> >>
> >>
> >> Here I add that we need to pay attention to the version of
> Scala and the
> >> version of JDK17
> >>
> >>
> >> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> [2] FLINK-30501 Update Flink build instruction to deprecate
> Java 8 instead
> >> of requiring Java 11
> https://issues.apache.org/jira/browse/FLINK-30501
> >>
> >> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四
> 00:54写道：
> >>
> >> > Hey dev community,
> >> >
> >> > I'm writing this email to kick off a discussion following
> this epic:
> >> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >> >
> >> > We are moving towards JDK 17 (LTS) , the only blocker now is
> Flink which
> >> > currently remains on JDK 11 (LTS). Flink does not support JDK
> 17 yet,
> >> with
> >> > no timeline, the reason, based on the aforementioned ticket
> is the
> >> > following tickets
> >> >
> >> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on
> Java 17<
> >> > https://issues.apache.org/jira/browse/FLINK-24998>.
> >> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest
> Kryo LTS
> >> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >> >
> >> > My question is whether it is possible to release a major
> version (Flink
> >> > 2.0.0) using the latest Kryo version for those who don't need
> to restore
> >> > old savepoints/checkpoints in newer format.
> >> >
> >> > 1. Leverage JDK 17 features within JVM
> >> > 2. Moving from the old format to the newer one will be
> handled only
> >> > once - a mitigation can be achieved by a conversion tool or
> external
> >> > serializers, both can be provided later on.
> >> >
> >> > I'd like to emphasize that the next JDK LTS (21) will be
> released this
> >> > September. furthermore, Flink already supports JDK 12-15,
> which is very
> >> > close to JDK 17 (LTS) - that was released in September 2021.
> JDK 11
> >> will
> >> > become a legacy soon, as more frameworks moving towards JDK
> 17 and are
> >> less
> >> > likely to support JDK 11 in the near future. (For example,
> Spring Boot 3
> >> > requires JDK 17 already).
> >> >
> >> > Thank you for your consideration of my request.
> >> >
> >> > Tamir.
> >> >
> >> >
> >> >
> >> >
> >> > Confidentiality: This communication and any attachments are
> intended for
> >> > the above-named persons only and may be confidential and/or
> legally
> >> > privileged. Any opinions expressed in this communication are not
> >> > necessarily those of NICE Actimize. If this communication has
> come to
> >> you
> >> > in error you must take no action based on it, nor must you
> copy or show
> >> it
> >> > to anyone; please delete/destroy and inform the sender by e-mail
> >> > immediately.
> >> > Monitoring: NICE Actimize may monitor incoming and outgoing
> e-mails.
> >> > Viruses: Although we have taken steps toward ensuring that
> this e-mail
> >> and
> >> > attachments are free from any virus, we advise that in
> keeping with good
> >> > computing practice the recipient should ensure they are
> actually virus
> >> free.
> >> >
> >>
> >>
> >> --
> >> Best
> >>
> >> ConradJam
> >>
> >> Confidentiality: This communication and any attachments are
> intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has
> come to you
> >> in error you must take no action based on it, nor must you copy
> or show it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing
> e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this
> e-mail
> >> and attachments are free from any virus, we advise that in
> keeping with
> >> good computing practice the recipient should ensure they are
> actually virus
> >> free.
> >>
> >>
> >>
> >>
> >> Confidentiality: This communication and any attachments are
> intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has
> come to you
> >> in error you must take no action based on it, nor must you copy
> or show it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing
> e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this
> e-mail
> >> and attachments are free from any virus, we advise that in
> keeping with
> >> good computing practice the recipient should ensure they are
> actually virus
> >> free.
> >>
> >
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Chesnay Schepler <ch...@apache.org>.
We don't know yet. I wanted to run some more experiments to see if I
cant get Scala 2.12.7 working on Java 17.
If that doesn't work, then it would also be an option to bump Scala in
the Java 17 builds (breaking savepoint compatibility), and users should
just only use the Java APIs.
The alternative to _that_ is doing this when we drop the Scala API.
On 28/04/2023 01:11, Thomas Weise wrote:
> Is the intention to bump the Flink major version and only support Java
> 17+? If so, can Scala not be upgraded at the same time?
>
> Thanks,
> Thomas
>
>
> On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser
> <ma...@apache.org> wrote:
>
> Scala 2.12.7 doesn't compile on Java 17, see
> https://issues.apache.org/jira/browse/FLINK-25000.
>
> On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>
> > Thanks Tamir for the information. According to the latest
> comment of the
> > task FLINK-24998, this bug should be gone while using the latest
> JDK 17. I
> > was wondering whether it means that there are no more issues to
> stop us
> > releasing a major Flink version to support Java 17? Did I miss
> something?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi
> <Ta...@niceactimize.com>
> > wrote:
> >
> >> More details about the JDK bug here
> >> https://bugs.openjdk.org/browse/JDK-8277529
> >>
> >> Related Jira ticket
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> ------------------------------
> >> *From:* Jing Ge via user <us...@flink.apache.org>
> >> *Sent:* Monday, April 24, 2023 11:15 PM
> >> *To:* Chesnay Schepler <ch...@apache.org>
> >> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis
> Sarda-Espinosa <
> >> sarda.espinosa@gmail.com>; Martijn Visser
> <ma...@apache.org>;
> >> dev@flink.apache.org <de...@flink.apache.org>; user
> <us...@flink.apache.org>
> >> *Subject:* Re: [Discussion] - Release major Flink version to
> support JDK
> >> 17 (LTS)
> >>
> >>
> >> *EXTERNAL EMAIL*
> >>
> >>
> >> Thanks Chesnay for working on this. Would you like to share
> more info
> >> about the JDK bug?
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler
> <ch...@apache.org>
> >> wrote:
> >>
> >> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
> >>
> >> On 31/03/2023 08:57, Chesnay Schepler wrote:
> >>
> >>
> >>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> >>
> >> Kroy themselves state that v5 likely can't read v2 data.
> >>
> >> However, both versions can be on the classpath without
> classpath as v5
> >> offers a versioned artifact that includes the version in the
> package.
> >>
> >> It probably wouldn't be difficult to migrate a savepoint to
> Kryo v5,
> >> purely from a read/write perspective.
> >>
> >> The bigger question is how we expose this new Kryo version in
> the API. If
> >> we stick to the versioned jar we need to either duplicate all
> current
> >> Kryo-related APIs or find a better way to integrate other
> serialization
> >> stacks.
> >> On 30/03/2023 17:50, Piotr Nowojski wrote:
> >>
> >> Hey,
> >>
> >> > 1. The Flink community agrees that we upgrade Kryo to a later
> version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala
> API support
> >> dropped. This is probably the quickest way, but would still
> mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we
> haven't been
> >> able to upgrade Kryo at all.
> >>
> >> This sounds pretty bad to me.
> >>
> >> Has anyone looked into what it would take to provide a smooth
> migration
> >> from Kryo2 -> Kryo5?
> >>
> >> Best,
> >> Piotrek
> >>
> >> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
> <sa...@gmail.com>
> >> napisał(a):
> >>
> >> Hi Martijn,
> >>
> >> just to be sure, if all state-related classes use a POJO
> serializer, Kryo
> >> will never come into play, right? Given FLINK-16686 [1], I
> wonder how many
> >> users actually have jobs with Kryo and RocksDB, but even if
> there aren't
> >> many, that still leaves those who don't use RocksDB for
> >> checkpoints/savepoints.
> >>
> >> If Kryo were to stay in the Flink APIs in v1.X, is it
> impossible to let
> >> users choose between v2/v5 jars by separating them like log4j2
> jars?
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-16686
> >>
> >> Regards,
> >> Alexis.
> >>
> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> >> martijnvisser@apache.org>:
> >>
> >> Hi all,
> >>
> >> I also saw a thread on this topic from Clayton Wohl [1] on this
> topic,
> >> which I'm including in this discussion thread to avoid that it
> gets lost.
> >>
> >> From my perspective, there's two main ways to get to Java 17:
> >>
> >> 1. The Flink community agrees that we upgrade Kryo to a later
> version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala
> API support
> >> dropped. This is probably the quickest way, but would still
> mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we
> haven't been
> >> able to upgrade Kryo at all.
> >> 2. There's a contributor who makes a contribution that bumps
> Kryo, but
> >> either a) automagically reads in all old checkpoints/savepoints
> in using
> >> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
> mentioned
> >> in the Kryo migration guide [2][3] or b) provides an offline
> tool that
> >> allows users that are interested in migrating their snapshots
> manually
> >> before starting from a newer version. That potentially could
> prevent the
> >> need to introduce a new Flink major version. In both scenarios,
> ideally the
> >> contributor would also help with avoiding the exposure of Kryo
> so that we
> >> will be in a better shape in the future.
> >>
> >> It would be good to get the opinion of the community for either
> of these
> >> two options, or potentially for another one that I haven't
> mentioned. If it
> >> appears that there's an overall agreement on the direction, I
> would propose
> >> that a FLIP gets created which describes the entire process.
> >>
> >> Looking forward to the thoughts of others, including the Users
> (therefore
> >> including the User ML).
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> [1]
> https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> >> [2]
> https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> >> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
> >>
> >> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
> <Ta...@niceactimize.com>
> >> wrote:
> >>
> >> I agree, there are several options to mitigate the migration
> from v2 to
> >> v5.
> >> yet, Oracle roadmap is to end JDK 11 support in September this
> year.
> >>
> >>
> >>
> >> ________________________________
> >> From: ConradJam <ja...@gmail.com>
> >> Sent: Thursday, March 16, 2023 4:36 AM
> >> To: dev@flink.apache.org <de...@flink.apache.org>
> >> Subject: Re: [Discussion] - Release major Flink version to
> support JDK 17
> >> (LTS)
> >>
> >> EXTERNAL EMAIL
> >>
> >>
> >>
> >> Thanks for your start this discuss
> >>
> >>
> >> I have been tracking this problem for a long time, until I saw a
> >> conversation in ISSUSE a few days ago and learned that the Kryo
> version
> >> problem will affect the JDK17 compilation of snapshots [1]
> FLINK-24998 ,
> >>
> >> As @cherry said it ruined our whole effort towards JDK17
> >>
> >> I am in favor of providing an external tool to migrate from
> Kryo old
> >> version checkpoint to the new Kryo new checkpoint at one time
> (Maybe this
> >> tool start in flink 2.0 ?), does this tool currently have any
> plans or
> >> ideas worth discuss
> >>
> >>
> >> I think it should not be difficult to be compatible with JDK11
> and JDK17.
> >> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in
> the doc
> >> that it is marked as Deprecated [2]
> >>
> >>
> >> Here I add that we need to pay attention to the version of
> Scala and the
> >> version of JDK17
> >>
> >>
> >> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> [2] FLINK-30501 Update Flink build instruction to deprecate
> Java 8 instead
> >> of requiring Java 11
> https://issues.apache.org/jira/browse/FLINK-30501
> >>
> >> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四
> 00:54写道：
> >>
> >> > Hey dev community,
> >> >
> >> > I'm writing this email to kick off a discussion following
> this epic:
> >> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >> >
> >> > We are moving towards JDK 17 (LTS) , the only blocker now is
> Flink which
> >> > currently remains on JDK 11 (LTS). Flink does not support JDK
> 17 yet,
> >> with
> >> > no timeline, the reason, based on the aforementioned ticket
> is the
> >> > following tickets
> >> >
> >> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on
> Java 17<
> >> > https://issues.apache.org/jira/browse/FLINK-24998>.
> >> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest
> Kryo LTS
> >> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >> >
> >> > My question is whether it is possible to release a major
> version (Flink
> >> > 2.0.0) using the latest Kryo version for those who don't need
> to restore
> >> > old savepoints/checkpoints in newer format.
> >> >
> >> > 1. Leverage JDK 17 features within JVM
> >> > 2. Moving from the old format to the newer one will be
> handled only
> >> > once - a mitigation can be achieved by a conversion tool or
> external
> >> > serializers, both can be provided later on.
> >> >
> >> > I'd like to emphasize that the next JDK LTS (21) will be
> released this
> >> > September. furthermore, Flink already supports JDK 12-15,
> which is very
> >> > close to JDK 17 (LTS) - that was released in September 2021.
> JDK 11
> >> will
> >> > become a legacy soon, as more frameworks moving towards JDK
> 17 and are
> >> less
> >> > likely to support JDK 11 in the near future. (For example,
> Spring Boot 3
> >> > requires JDK 17 already).
> >> >
> >> > Thank you for your consideration of my request.
> >> >
> >> > Tamir.
> >> >
> >> >
> >> >
> >> >
> >> > Confidentiality: This communication and any attachments are
> intended for
> >> > the above-named persons only and may be confidential and/or
> legally
> >> > privileged. Any opinions expressed in this communication are not
> >> > necessarily those of NICE Actimize. If this communication has
> come to
> >> you
> >> > in error you must take no action based on it, nor must you
> copy or show
> >> it
> >> > to anyone; please delete/destroy and inform the sender by e-mail
> >> > immediately.
> >> > Monitoring: NICE Actimize may monitor incoming and outgoing
> e-mails.
> >> > Viruses: Although we have taken steps toward ensuring that
> this e-mail
> >> and
> >> > attachments are free from any virus, we advise that in
> keeping with good
> >> > computing practice the recipient should ensure they are
> actually virus
> >> free.
> >> >
> >>
> >>
> >> --
> >> Best
> >>
> >> ConradJam
> >>
> >> Confidentiality: This communication and any attachments are
> intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has
> come to you
> >> in error you must take no action based on it, nor must you copy
> or show it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing
> e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this
> e-mail
> >> and attachments are free from any virus, we advise that in
> keeping with
> >> good computing practice the recipient should ensure they are
> actually virus
> >> free.
> >>
> >>
> >>
> >>
> >> Confidentiality: This communication and any attachments are
> intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has
> come to you
> >> in error you must take no action based on it, nor must you copy
> or show it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing
> e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this
> e-mail
> >> and attachments are free from any virus, we advise that in
> keeping with
> >> good computing practice the recipient should ensure they are
> actually virus
> >> free.
> >>
> >
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Thomas Weise <th...@apache.org>.
Is the intention to bump the Flink major version and only support Java 17+?
If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser <ma...@apache.org>
wrote:
> Scala 2.12.7 doesn't compile on Java 17, see
> https://issues.apache.org/jira/browse/FLINK-25000.
>
> On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>
> > Thanks Tamir for the information. According to the latest comment of the
> > task FLINK-24998, this bug should be gone while using the latest JDK 17.
> I
> > was wondering whether it means that there are no more issues to stop us
> > releasing a major Flink version to support Java 17? Did I miss something?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>
> > wrote:
> >
> >> More details about the JDK bug here
> >> https://bugs.openjdk.org/browse/JDK-8277529
> >>
> >> Related Jira ticket
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> ------------------------------
> >> *From:* Jing Ge via user <us...@flink.apache.org>
> >> *Sent:* Monday, April 24, 2023 11:15 PM
> >> *To:* Chesnay Schepler <ch...@apache.org>
> >> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
> >> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
> >> dev@flink.apache.org <de...@flink.apache.org>; user <
> user@flink.apache.org>
> >> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
> >> 17 (LTS)
> >>
> >>
> >> *EXTERNAL EMAIL*
> >>
> >>
> >> Thanks Chesnay for working on this. Would you like to share more info
> >> about the JDK bug?
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
> >> wrote:
> >>
> >> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
> >>
> >> On 31/03/2023 08:57, Chesnay Schepler wrote:
> >>
> >>
> >>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> >>
> >> Kroy themselves state that v5 likely can't read v2 data.
> >>
> >> However, both versions can be on the classpath without classpath as v5
> >> offers a versioned artifact that includes the version in the package.
> >>
> >> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> >> purely from a read/write perspective.
> >>
> >> The bigger question is how we expose this new Kryo version in the API.
> If
> >> we stick to the versioned jar we need to either duplicate all current
> >> Kryo-related APIs or find a better way to integrate other serialization
> >> stacks.
> >> On 30/03/2023 17:50, Piotr Nowojski wrote:
> >>
> >> Hey,
> >>
> >> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
> support
> >> dropped. This is probably the quickest way, but would still mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
> been
> >> able to upgrade Kryo at all.
> >>
> >> This sounds pretty bad to me.
> >>
> >> Has anyone looked into what it would take to provide a smooth migration
> >> from Kryo2 -> Kryo5?
> >>
> >> Best,
> >> Piotrek
> >>
> >> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com>
> >> napisał(a):
> >>
> >> Hi Martijn,
> >>
> >> just to be sure, if all state-related classes use a POJO serializer,
> Kryo
> >> will never come into play, right? Given FLINK-16686 [1], I wonder how
> many
> >> users actually have jobs with Kryo and RocksDB, but even if there aren't
> >> many, that still leaves those who don't use RocksDB for
> >> checkpoints/savepoints.
> >>
> >> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
> >> users choose between v2/v5 jars by separating them like log4j2 jars?
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-16686
> >>
> >> Regards,
> >> Alexis.
> >>
> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> >> martijnvisser@apache.org>:
> >>
> >> Hi all,
> >>
> >> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
> >> which I'm including in this discussion thread to avoid that it gets
> lost.
> >>
> >> From my perspective, there's two main ways to get to Java 17:
> >>
> >> 1. The Flink community agrees that we upgrade Kryo to a later version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
> support
> >> dropped. This is probably the quickest way, but would still mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
> been
> >> able to upgrade Kryo at all.
> >> 2. There's a contributor who makes a contribution that bumps Kryo, but
> >> either a) automagically reads in all old checkpoints/savepoints in using
> >> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
> mentioned
> >> in the Kryo migration guide [2][3] or b) provides an offline tool that
> >> allows users that are interested in migrating their snapshots manually
> >> before starting from a newer version. That potentially could prevent the
> >> need to introduce a new Flink major version. In both scenarios, ideally
> the
> >> contributor would also help with avoiding the exposure of Kryo so that
> we
> >> will be in a better shape in the future.
> >>
> >> It would be good to get the opinion of the community for either of these
> >> two options, or potentially for another one that I haven't mentioned.
> If it
> >> appears that there's an overall agreement on the direction, I would
> propose
> >> that a FLIP gets created which describes the entire process.
> >>
> >> Looking forward to the thoughts of others, including the Users
> (therefore
> >> including the User ML).
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> >> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> >> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
> >>
> >> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Tamir.Sagi@niceactimize.com
> >
> >> wrote:
> >>
> >> I agree, there are several options to mitigate the migration from v2 to
> >> v5.
> >> yet, Oracle roadmap is to end JDK 11 support in September this year.
> >>
> >>
> >>
> >> ________________________________
> >> From: ConradJam <ja...@gmail.com>
> >> Sent: Thursday, March 16, 2023 4:36 AM
> >> To: dev@flink.apache.org <de...@flink.apache.org>
> >> Subject: Re: [Discussion] - Release major Flink version to support JDK
> 17
> >> (LTS)
> >>
> >> EXTERNAL EMAIL
> >>
> >>
> >>
> >> Thanks for your start this discuss
> >>
> >>
> >> I have been tracking this problem for a long time, until I saw a
> >> conversation in ISSUSE a few days ago and learned that the Kryo version
> >> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
> >>
> >> As @cherry said it ruined our whole effort towards JDK17
> >>
> >> I am in favor of providing an external tool to migrate from Kryo old
> >> version checkpoint to the new Kryo new checkpoint at one time (Maybe
> this
> >> tool start in flink 2.0 ?), does this tool currently have any plans or
> >> ideas worth discuss
> >>
> >>
> >> I think it should not be difficult to be compatible with JDK11 and
> JDK17.
> >> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
> >> that it is marked as Deprecated [2]
> >>
> >>
> >> Here I add that we need to pay attention to the version of Scala and the
> >> version of JDK17
> >>
> >>
> >> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
> instead
> >> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
> >>
> >> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
> >>
> >> > Hey dev community,
> >> >
> >> > I'm writing this email to kick off a discussion following this epic:
> >> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >> >
> >> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
> which
> >> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
> >> with
> >> > no timeline, the reason, based on the aforementioned ticket is the
> >> > following tickets
> >> >
> >> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> >> > https://issues.apache.org/jira/browse/FLINK-24998>.
> >> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> >> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >> >
> >> > My question is whether it is possible to release a major version
> (Flink
> >> > 2.0.0) using the latest Kryo version for those who don't need to
> restore
> >> > old savepoints/checkpoints in newer format.
> >> >
> >> > 1. Leverage JDK 17 features within JVM
> >> > 2. Moving from the old format to the newer one will be handled only
> >> > once - a mitigation can be achieved by a conversion tool or external
> >> > serializers, both can be provided later on.
> >> >
> >> > I'd like to emphasize that the next JDK LTS (21) will be released this
> >> > September. furthermore, Flink already supports JDK 12-15, which is
> very
> >> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
> >> will
> >> > become a legacy soon, as more frameworks moving towards JDK 17 and are
> >> less
> >> > likely to support JDK 11 in the near future. (For example, Spring
> Boot 3
> >> > requires JDK 17 already).
> >> >
> >> > Thank you for your consideration of my request.
> >> >
> >> > Tamir.
> >> >
> >> >
> >> >
> >> >
> >> > Confidentiality: This communication and any attachments are intended
> for
> >> > the above-named persons only and may be confidential and/or legally
> >> > privileged. Any opinions expressed in this communication are not
> >> > necessarily those of NICE Actimize. If this communication has come to
> >> you
> >> > in error you must take no action based on it, nor must you copy or
> show
> >> it
> >> > to anyone; please delete/destroy and inform the sender by e-mail
> >> > immediately.
> >> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> >> > Viruses: Although we have taken steps toward ensuring that this e-mail
> >> and
> >> > attachments are free from any virus, we advise that in keeping with
> good
> >> > computing practice the recipient should ensure they are actually virus
> >> free.
> >> >
> >>
> >>
> >> --
> >> Best
> >>
> >> ConradJam
> >>
> >> Confidentiality: This communication and any attachments are intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has come to
> you
> >> in error you must take no action based on it, nor must you copy or show
> it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this e-mail
> >> and attachments are free from any virus, we advise that in keeping with
> >> good computing practice the recipient should ensure they are actually
> virus
> >> free.
> >>
> >>
> >>
> >>
> >> Confidentiality: This communication and any attachments are intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has come to
> you
> >> in error you must take no action based on it, nor must you copy or show
> it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this e-mail
> >> and attachments are free from any virus, we advise that in keeping with
> >> good computing practice the recipient should ensure they are actually
> virus
> >> free.
> >>
> >
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Thomas Weise <th...@apache.org>.
Is the intention to bump the Flink major version and only support Java 17+?
If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser <ma...@apache.org>
wrote:
> Scala 2.12.7 doesn't compile on Java 17, see
> https://issues.apache.org/jira/browse/FLINK-25000.
>
> On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>
> > Thanks Tamir for the information. According to the latest comment of the
> > task FLINK-24998, this bug should be gone while using the latest JDK 17.
> I
> > was wondering whether it means that there are no more issues to stop us
> > releasing a major Flink version to support Java 17? Did I miss something?
> >
> > Best regards,
> > Jing
> >
> > On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>
> > wrote:
> >
> >> More details about the JDK bug here
> >> https://bugs.openjdk.org/browse/JDK-8277529
> >>
> >> Related Jira ticket
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> ------------------------------
> >> *From:* Jing Ge via user <us...@flink.apache.org>
> >> *Sent:* Monday, April 24, 2023 11:15 PM
> >> *To:* Chesnay Schepler <ch...@apache.org>
> >> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
> >> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
> >> dev@flink.apache.org <de...@flink.apache.org>; user <
> user@flink.apache.org>
> >> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
> >> 17 (LTS)
> >>
> >>
> >> *EXTERNAL EMAIL*
> >>
> >>
> >> Thanks Chesnay for working on this. Would you like to share more info
> >> about the JDK bug?
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
> >> wrote:
> >>
> >> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
> >>
> >> On 31/03/2023 08:57, Chesnay Schepler wrote:
> >>
> >>
> >>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
> >>
> >> Kroy themselves state that v5 likely can't read v2 data.
> >>
> >> However, both versions can be on the classpath without classpath as v5
> >> offers a versioned artifact that includes the version in the package.
> >>
> >> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> >> purely from a read/write perspective.
> >>
> >> The bigger question is how we expose this new Kryo version in the API.
> If
> >> we stick to the versioned jar we need to either duplicate all current
> >> Kryo-related APIs or find a better way to integrate other serialization
> >> stacks.
> >> On 30/03/2023 17:50, Piotr Nowojski wrote:
> >>
> >> Hey,
> >>
> >> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
> support
> >> dropped. This is probably the quickest way, but would still mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
> been
> >> able to upgrade Kryo at all.
> >>
> >> This sounds pretty bad to me.
> >>
> >> Has anyone looked into what it would take to provide a smooth migration
> >> from Kryo2 -> Kryo5?
> >>
> >> Best,
> >> Piotrek
> >>
> >> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com>
> >> napisał(a):
> >>
> >> Hi Martijn,
> >>
> >> just to be sure, if all state-related classes use a POJO serializer,
> Kryo
> >> will never come into play, right? Given FLINK-16686 [1], I wonder how
> many
> >> users actually have jobs with Kryo and RocksDB, but even if there aren't
> >> many, that still leaves those who don't use RocksDB for
> >> checkpoints/savepoints.
> >>
> >> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
> >> users choose between v2/v5 jars by separating them like log4j2 jars?
> >>
> >> [1] https://issues.apache.org/jira/browse/FLINK-16686
> >>
> >> Regards,
> >> Alexis.
> >>
> >> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> >> martijnvisser@apache.org>:
> >>
> >> Hi all,
> >>
> >> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
> >> which I'm including in this discussion thread to avoid that it gets
> lost.
> >>
> >> From my perspective, there's two main ways to get to Java 17:
> >>
> >> 1. The Flink community agrees that we upgrade Kryo to a later version,
> >> which means breaking all checkpoint/savepoint compatibility and
> releasing a
> >> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API
> support
> >> dropped. This is probably the quickest way, but would still mean that we
> >> expose Kryo in the Flink APIs, which is the main reason why we haven't
> been
> >> able to upgrade Kryo at all.
> >> 2. There's a contributor who makes a contribution that bumps Kryo, but
> >> either a) automagically reads in all old checkpoints/savepoints in using
> >> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
> mentioned
> >> in the Kryo migration guide [2][3] or b) provides an offline tool that
> >> allows users that are interested in migrating their snapshots manually
> >> before starting from a newer version. That potentially could prevent the
> >> need to introduce a new Flink major version. In both scenarios, ideally
> the
> >> contributor would also help with avoiding the exposure of Kryo so that
> we
> >> will be in a better shape in the future.
> >>
> >> It would be good to get the opinion of the community for either of these
> >> two options, or potentially for another one that I haven't mentioned.
> If it
> >> appears that there's an overall agreement on the direction, I would
> propose
> >> that a FLIP gets created which describes the entire process.
> >>
> >> Looking forward to the thoughts of others, including the Users
> (therefore
> >> including the User ML).
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> >> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> >> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
> >>
> >> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Tamir.Sagi@niceactimize.com
> >
> >> wrote:
> >>
> >> I agree, there are several options to mitigate the migration from v2 to
> >> v5.
> >> yet, Oracle roadmap is to end JDK 11 support in September this year.
> >>
> >>
> >>
> >> ________________________________
> >> From: ConradJam <ja...@gmail.com>
> >> Sent: Thursday, March 16, 2023 4:36 AM
> >> To: dev@flink.apache.org <de...@flink.apache.org>
> >> Subject: Re: [Discussion] - Release major Flink version to support JDK
> 17
> >> (LTS)
> >>
> >> EXTERNAL EMAIL
> >>
> >>
> >>
> >> Thanks for your start this discuss
> >>
> >>
> >> I have been tracking this problem for a long time, until I saw a
> >> conversation in ISSUSE a few days ago and learned that the Kryo version
> >> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
> >>
> >> As @cherry said it ruined our whole effort towards JDK17
> >>
> >> I am in favor of providing an external tool to migrate from Kryo old
> >> version checkpoint to the new Kryo new checkpoint at one time (Maybe
> this
> >> tool start in flink 2.0 ?), does this tool currently have any plans or
> >> ideas worth discuss
> >>
> >>
> >> I think it should not be difficult to be compatible with JDK11 and
> JDK17.
> >> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
> >> that it is marked as Deprecated [2]
> >>
> >>
> >> Here I add that we need to pay attention to the version of Scala and the
> >> version of JDK17
> >>
> >>
> >> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> >> https://issues.apache.org/jira/browse/FLINK-24998
> >>
> >> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
> instead
> >> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
> >>
> >> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
> >>
> >> > Hey dev community,
> >> >
> >> > I'm writing this email to kick off a discussion following this epic:
> >> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >> >
> >> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
> which
> >> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
> >> with
> >> > no timeline, the reason, based on the aforementioned ticket is the
> >> > following tickets
> >> >
> >> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> >> > https://issues.apache.org/jira/browse/FLINK-24998>.
> >> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> >> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >> >
> >> > My question is whether it is possible to release a major version
> (Flink
> >> > 2.0.0) using the latest Kryo version for those who don't need to
> restore
> >> > old savepoints/checkpoints in newer format.
> >> >
> >> > 1. Leverage JDK 17 features within JVM
> >> > 2. Moving from the old format to the newer one will be handled only
> >> > once - a mitigation can be achieved by a conversion tool or external
> >> > serializers, both can be provided later on.
> >> >
> >> > I'd like to emphasize that the next JDK LTS (21) will be released this
> >> > September. furthermore, Flink already supports JDK 12-15, which is
> very
> >> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
> >> will
> >> > become a legacy soon, as more frameworks moving towards JDK 17 and are
> >> less
> >> > likely to support JDK 11 in the near future. (For example, Spring
> Boot 3
> >> > requires JDK 17 already).
> >> >
> >> > Thank you for your consideration of my request.
> >> >
> >> > Tamir.
> >> >
> >> >
> >> >
> >> >
> >> > Confidentiality: This communication and any attachments are intended
> for
> >> > the above-named persons only and may be confidential and/or legally
> >> > privileged. Any opinions expressed in this communication are not
> >> > necessarily those of NICE Actimize. If this communication has come to
> >> you
> >> > in error you must take no action based on it, nor must you copy or
> show
> >> it
> >> > to anyone; please delete/destroy and inform the sender by e-mail
> >> > immediately.
> >> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> >> > Viruses: Although we have taken steps toward ensuring that this e-mail
> >> and
> >> > attachments are free from any virus, we advise that in keeping with
> good
> >> > computing practice the recipient should ensure they are actually virus
> >> free.
> >> >
> >>
> >>
> >> --
> >> Best
> >>
> >> ConradJam
> >>
> >> Confidentiality: This communication and any attachments are intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has come to
> you
> >> in error you must take no action based on it, nor must you copy or show
> it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this e-mail
> >> and attachments are free from any virus, we advise that in keeping with
> >> good computing practice the recipient should ensure they are actually
> virus
> >> free.
> >>
> >>
> >>
> >>
> >> Confidentiality: This communication and any attachments are intended for
> >> the above-named persons only and may be confidential and/or legally
> >> privileged. Any opinions expressed in this communication are not
> >> necessarily those of NICE Actimize. If this communication has come to
> you
> >> in error you must take no action based on it, nor must you copy or show
> it
> >> to anyone; please delete/destroy and inform the sender by e-mail
> >> immediately.
> >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> >> Viruses: Although we have taken steps toward ensuring that this e-mail
> >> and attachments are free from any virus, we advise that in keeping with
> >> good computing practice the recipient should ensure they are actually
> virus
> >> free.
> >>
> >
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Martijn Visser <ma...@apache.org>.
Scala 2.12.7 doesn't compile on Java 17, see
https://issues.apache.org/jira/browse/FLINK-25000.
On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
> Thanks Tamir for the information. According to the latest comment of the
> task FLINK-24998, this bug should be gone while using the latest JDK 17. I
> was wondering whether it means that there are no more issues to stop us
> releasing a major Flink version to support Java 17? Did I miss something?
>
> Best regards,
> Jing
>
> On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
>
>> More details about the JDK bug here
>> https://bugs.openjdk.org/browse/JDK-8277529
>>
>> Related Jira ticket
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> ------------------------------
>> *From:* Jing Ge via user <us...@flink.apache.org>
>> *Sent:* Monday, April 24, 2023 11:15 PM
>> *To:* Chesnay Schepler <ch...@apache.org>
>> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
>> dev@flink.apache.org <de...@flink.apache.org>; user <us...@flink.apache.org>
>> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
>> 17 (LTS)
>>
>>
>> *EXTERNAL EMAIL*
>>
>>
>> Thanks Chesnay for working on this. Would you like to share more info
>> about the JDK bug?
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>>
>> On 31/03/2023 08:57, Chesnay Schepler wrote:
>>
>>
>> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>>
>> Kroy themselves state that v5 likely can't read v2 data.
>>
>> However, both versions can be on the classpath without classpath as v5
>> offers a versioned artifact that includes the version in the package.
>>
>> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
>> purely from a read/write perspective.
>>
>> The bigger question is how we expose this new Kryo version in the API. If
>> we stick to the versioned jar we need to either duplicate all current
>> Kryo-related APIs or find a better way to integrate other serialization
>> stacks.
>> On 30/03/2023 17:50, Piotr Nowojski wrote:
>>
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth migration
>> from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
>> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvisser@apache.org>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> which I'm including in this discussion thread to avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots manually
>> before starting from a newer version. That potentially could prevent the
>> need to introduce a new Flink major version. In both scenarios, ideally the
>> contributor would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either of these
>> two options, or potentially for another one that I haven't mentioned. If it
>> appears that there's an overall agreement on the direction, I would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users (therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
>> wrote:
>>
>> I agree, there are several options to mitigate the migration from v2 to
>> v5.
>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
>> (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until I saw a
>> conversation in ISSUSE a few days ago and learned that the Kryo version
>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion following this epic:
>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> with
>> > no timeline, the reason, based on the aforementioned ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a major version (Flink
>> > 2.0.0) using the latest Kryo version for those who don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will be handled only
>> > once - a mitigation can be achieved by a conversion tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>> > September. furthermore, Flink already supports JDK 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> will
>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>> less
>> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments are intended for
>> > the above-named persons only and may be confidential and/or legally
>> > privileged. Any opinions expressed in this communication are not
>> > necessarily those of NICE Actimize. If this communication has come to
>> you
>> > in error you must take no action based on it, nor must you copy or show
>> it
>> > to anyone; please delete/destroy and inform the sender by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>> and
>> > attachments are free from any virus, we advise that in keeping with good
>> > computing practice the recipient should ensure they are actually virus
>> free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>>
>>
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Martijn Visser <ma...@apache.org>.
Scala 2.12.7 doesn't compile on Java 17, see
https://issues.apache.org/jira/browse/FLINK-25000.
On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
> Thanks Tamir for the information. According to the latest comment of the
> task FLINK-24998, this bug should be gone while using the latest JDK 17. I
> was wondering whether it means that there are no more issues to stop us
> releasing a major Flink version to support Java 17? Did I miss something?
>
> Best regards,
> Jing
>
> On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
>
>> More details about the JDK bug here
>> https://bugs.openjdk.org/browse/JDK-8277529
>>
>> Related Jira ticket
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> ------------------------------
>> *From:* Jing Ge via user <us...@flink.apache.org>
>> *Sent:* Monday, April 24, 2023 11:15 PM
>> *To:* Chesnay Schepler <ch...@apache.org>
>> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
>> dev@flink.apache.org <de...@flink.apache.org>; user <us...@flink.apache.org>
>> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
>> 17 (LTS)
>>
>>
>> *EXTERNAL EMAIL*
>>
>>
>> Thanks Chesnay for working on this. Would you like to share more info
>> about the JDK bug?
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>>
>> On 31/03/2023 08:57, Chesnay Schepler wrote:
>>
>>
>> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>>
>> Kroy themselves state that v5 likely can't read v2 data.
>>
>> However, both versions can be on the classpath without classpath as v5
>> offers a versioned artifact that includes the version in the package.
>>
>> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
>> purely from a read/write perspective.
>>
>> The bigger question is how we expose this new Kryo version in the API. If
>> we stick to the versioned jar we need to either duplicate all current
>> Kryo-related APIs or find a better way to integrate other serialization
>> stacks.
>> On 30/03/2023 17:50, Piotr Nowojski wrote:
>>
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth migration
>> from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
>> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvisser@apache.org>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> which I'm including in this discussion thread to avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots manually
>> before starting from a newer version. That potentially could prevent the
>> need to introduce a new Flink major version. In both scenarios, ideally the
>> contributor would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either of these
>> two options, or potentially for another one that I haven't mentioned. If it
>> appears that there's an overall agreement on the direction, I would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users (therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
>> wrote:
>>
>> I agree, there are several options to mitigate the migration from v2 to
>> v5.
>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
>> (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until I saw a
>> conversation in ISSUSE a few days ago and learned that the Kryo version
>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion following this epic:
>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> with
>> > no timeline, the reason, based on the aforementioned ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a major version (Flink
>> > 2.0.0) using the latest Kryo version for those who don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will be handled only
>> > once - a mitigation can be achieved by a conversion tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>> > September. furthermore, Flink already supports JDK 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> will
>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>> less
>> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments are intended for
>> > the above-named persons only and may be confidential and/or legally
>> > privileged. Any opinions expressed in this communication are not
>> > necessarily those of NICE Actimize. If this communication has come to
>> you
>> > in error you must take no action based on it, nor must you copy or show
>> it
>> > to anyone; please delete/destroy and inform the sender by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>> and
>> > attachments are free from any virus, we advise that in keeping with good
>> > computing practice the recipient should ensure they are actually virus
>> free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>>
>>
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Thanks Tamir for the information. According to the latest comment of the
task FLINK-24998, this bug should be gone while using the latest JDK 17. I
was wondering whether it means that there are no more issues to stop us
releasing a major Flink version to support Java 17? Did I miss something?
Best regards,
Jing
On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>
wrote:
> More details about the JDK bug here
> https://bugs.openjdk.org/browse/JDK-8277529
>
> Related Jira ticket
> https://issues.apache.org/jira/browse/FLINK-24998
>
> ------------------------------
> *From:* Jing Ge via user <us...@flink.apache.org>
> *Sent:* Monday, April 24, 2023 11:15 PM
> *To:* Chesnay Schepler <ch...@apache.org>
> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
> dev@flink.apache.org <de...@flink.apache.org>; user <us...@flink.apache.org>
> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
> 17 (LTS)
>
>
> *EXTERNAL EMAIL*
>
>
> Thanks Chesnay for working on this. Would you like to share more info
> about the JDK bug?
>
> Best regards,
> Jing
>
> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>
> On 31/03/2023 08:57, Chesnay Schepler wrote:
>
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API. If
> we stick to the versioned jar we need to either duplicate all current
> Kryo-related APIs or find a better way to integrate other serialization
> stacks.
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth migration
> from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
> napisał(a):
>
> Hi Martijn,
>
> just to be sure, if all state-related classes use a POJO serializer, Kryo
> will never come into play, right? Given FLINK-16686 [1], I wonder how many
> users actually have jobs with Kryo and RocksDB, but even if there aren't
> many, that still leaves those who don't use RocksDB for
> checkpoints/savepoints.
>
> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
> users choose between v2/v5 jars by separating them like log4j2 jars?
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
> Regards,
> Alexis.
>
> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> martijnvisser@apache.org>:
>
> Hi all,
>
> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
> which I'm including in this discussion thread to avoid that it gets lost.
>
> From my perspective, there's two main ways to get to Java 17:
>
> 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
> 2. There's a contributor who makes a contribution that bumps Kryo, but
> either a) automagically reads in all old checkpoints/savepoints in using
> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
> in the Kryo migration guide [2][3] or b) provides an offline tool that
> allows users that are interested in migrating their snapshots manually
> before starting from a newer version. That potentially could prevent the
> need to introduce a new Flink major version. In both scenarios, ideally the
> contributor would also help with avoiding the exposure of Kryo so that we
> will be in a better shape in the future.
>
> It would be good to get the opinion of the community for either of these
> two options, or potentially for another one that I haven't mentioned. If it
> appears that there's an overall agreement on the direction, I would propose
> that a FLIP gets created which describes the entire process.
>
> Looking forward to the thoughts of others, including the Users (therefore
> including the User ML).
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>
> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
>
> I agree, there are several options to mitigate the migration from v2 to v5.
> yet, Oracle roadmap is to end JDK 11 support in September this year.
>
>
>
> ________________________________
> From: ConradJam <ja...@gmail.com>
> Sent: Thursday, March 16, 2023 4:36 AM
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
> (LTS)
>
> EXTERNAL EMAIL
>
>
>
> Thanks for your start this discuss
>
>
> I have been tracking this problem for a long time, until I saw a
> conversation in ISSUSE a few days ago and learned that the Kryo version
> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>
> As @cherry said it ruined our whole effort towards JDK17
>
> I am in favor of providing an external tool to migrate from Kryo old
> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
> tool start in flink 2.0 ?), does this tool currently have any plans or
> ideas worth discuss
>
>
> I think it should not be difficult to be compatible with JDK11 and JDK17.
> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
> that it is marked as Deprecated [2]
>
>
> Here I add that we need to pay attention to the version of Scala and the
> version of JDK17
>
>
> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> https://issues.apache.org/jira/browse/FLINK-24998
>
> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>
> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>
> > Hey dev community,
> >
> > I'm writing this email to kick off a discussion following this epic:
> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >
> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
> with
> > no timeline, the reason, based on the aforementioned ticket is the
> > following tickets
> >
> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> > https://issues.apache.org/jira/browse/FLINK-24998>.
> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >
> > My question is whether it is possible to release a major version (Flink
> > 2.0.0) using the latest Kryo version for those who don't need to restore
> > old savepoints/checkpoints in newer format.
> >
> > 1. Leverage JDK 17 features within JVM
> > 2. Moving from the old format to the newer one will be handled only
> > once - a mitigation can be achieved by a conversion tool or external
> > serializers, both can be provided later on.
> >
> > I'd like to emphasize that the next JDK LTS (21) will be released this
> > September. furthermore, Flink already supports JDK 12-15, which is very
> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11 will
> > become a legacy soon, as more frameworks moving towards JDK 17 and are
> less
> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
> > requires JDK 17 already).
> >
> > Thank you for your consideration of my request.
> >
> > Tamir.
> >
> >
> >
> >
> > Confidentiality: This communication and any attachments are intended for
> > the above-named persons only and may be confidential and/or legally
> > privileged. Any opinions expressed in this communication are not
> > necessarily those of NICE Actimize. If this communication has come to you
> > in error you must take no action based on it, nor must you copy or show
> it
> > to anyone; please delete/destroy and inform the sender by e-mail
> > immediately.
> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring that this e-mail
> and
> > attachments are free from any virus, we advise that in keeping with good
> > computing practice the recipient should ensure they are actually virus
> free.
> >
>
>
> --
> Best
>
> ConradJam
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
>
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Jing Ge via user <us...@flink.apache.org>.
Thanks Tamir for the information. According to the latest comment of the
task FLINK-24998, this bug should be gone while using the latest JDK 17. I
was wondering whether it means that there are no more issues to stop us
releasing a major Flink version to support Java 17? Did I miss something?
Best regards,
Jing
On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi <Ta...@niceactimize.com>
wrote:
> More details about the JDK bug here
> https://bugs.openjdk.org/browse/JDK-8277529
>
> Related Jira ticket
> https://issues.apache.org/jira/browse/FLINK-24998
>
> ------------------------------
> *From:* Jing Ge via user <us...@flink.apache.org>
> *Sent:* Monday, April 24, 2023 11:15 PM
> *To:* Chesnay Schepler <ch...@apache.org>
> *Cc:* Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com>; Martijn Visser <ma...@apache.org>;
> dev@flink.apache.org <de...@flink.apache.org>; user <us...@flink.apache.org>
> *Subject:* Re: [Discussion] - Release major Flink version to support JDK
> 17 (LTS)
>
>
> *EXTERNAL EMAIL*
>
>
> Thanks Chesnay for working on this. Would you like to share more info
> about the JDK bug?
>
> Best regards,
> Jing
>
> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>
> On 31/03/2023 08:57, Chesnay Schepler wrote:
>
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API. If
> we stick to the versioned jar we need to either duplicate all current
> Kryo-related APIs or find a better way to integrate other serialization
> stacks.
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth migration
> from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
> napisał(a):
>
> Hi Martijn,
>
> just to be sure, if all state-related classes use a POJO serializer, Kryo
> will never come into play, right? Given FLINK-16686 [1], I wonder how many
> users actually have jobs with Kryo and RocksDB, but even if there aren't
> many, that still leaves those who don't use RocksDB for
> checkpoints/savepoints.
>
> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
> users choose between v2/v5 jars by separating them like log4j2 jars?
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
> Regards,
> Alexis.
>
> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> martijnvisser@apache.org>:
>
> Hi all,
>
> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
> which I'm including in this discussion thread to avoid that it gets lost.
>
> From my perspective, there's two main ways to get to Java 17:
>
> 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
> 2. There's a contributor who makes a contribution that bumps Kryo, but
> either a) automagically reads in all old checkpoints/savepoints in using
> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
> in the Kryo migration guide [2][3] or b) provides an offline tool that
> allows users that are interested in migrating their snapshots manually
> before starting from a newer version. That potentially could prevent the
> need to introduce a new Flink major version. In both scenarios, ideally the
> contributor would also help with avoiding the exposure of Kryo so that we
> will be in a better shape in the future.
>
> It would be good to get the opinion of the community for either of these
> two options, or potentially for another one that I haven't mentioned. If it
> appears that there's an overall agreement on the direction, I would propose
> that a FLIP gets created which describes the entire process.
>
> Looking forward to the thoughts of others, including the Users (therefore
> including the User ML).
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>
> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
>
> I agree, there are several options to mitigate the migration from v2 to v5.
> yet, Oracle roadmap is to end JDK 11 support in September this year.
>
>
>
> ________________________________
> From: ConradJam <ja...@gmail.com>
> Sent: Thursday, March 16, 2023 4:36 AM
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
> (LTS)
>
> EXTERNAL EMAIL
>
>
>
> Thanks for your start this discuss
>
>
> I have been tracking this problem for a long time, until I saw a
> conversation in ISSUSE a few days ago and learned that the Kryo version
> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>
> As @cherry said it ruined our whole effort towards JDK17
>
> I am in favor of providing an external tool to migrate from Kryo old
> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
> tool start in flink 2.0 ?), does this tool currently have any plans or
> ideas worth discuss
>
>
> I think it should not be difficult to be compatible with JDK11 and JDK17.
> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
> that it is marked as Deprecated [2]
>
>
> Here I add that we need to pay attention to the version of Scala and the
> version of JDK17
>
>
> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> https://issues.apache.org/jira/browse/FLINK-24998
>
> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>
> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>
> > Hey dev community,
> >
> > I'm writing this email to kick off a discussion following this epic:
> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >
> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
> with
> > no timeline, the reason, based on the aforementioned ticket is the
> > following tickets
> >
> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> > https://issues.apache.org/jira/browse/FLINK-24998>.
> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >
> > My question is whether it is possible to release a major version (Flink
> > 2.0.0) using the latest Kryo version for those who don't need to restore
> > old savepoints/checkpoints in newer format.
> >
> > 1. Leverage JDK 17 features within JVM
> > 2. Moving from the old format to the newer one will be handled only
> > once - a mitigation can be achieved by a conversion tool or external
> > serializers, both can be provided later on.
> >
> > I'd like to emphasize that the next JDK LTS (21) will be released this
> > September. furthermore, Flink already supports JDK 12-15, which is very
> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11 will
> > become a legacy soon, as more frameworks moving towards JDK 17 and are
> less
> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
> > requires JDK 17 already).
> >
> > Thank you for your consideration of my request.
> >
> > Tamir.
> >
> >
> >
> >
> > Confidentiality: This communication and any attachments are intended for
> > the above-named persons only and may be confidential and/or legally
> > privileged. Any opinions expressed in this communication are not
> > necessarily those of NICE Actimize. If this communication has come to you
> > in error you must take no action based on it, nor must you copy or show
> it
> > to anyone; please delete/destroy and inform the sender by e-mail
> > immediately.
> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring that this e-mail
> and
> > attachments are free from any virus, we advise that in keeping with good
> > computing practice the recipient should ensure they are actually virus
> free.
> >
>
>
> --
> Best
>
> ConradJam
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
>
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Tamir Sagi <Ta...@niceactimize.com>.
More details about the JDK bug here
https://bugs.openjdk.org/browse/JDK-8277529
Related Jira ticket
https://issues.apache.org/jira/browse/FLINK-24998
________________________________
From: Jing Ge via user <us...@flink.apache.org>
Sent: Monday, April 24, 2023 11:15 PM
To: Chesnay Schepler <ch...@apache.org>
Cc: Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <sa...@gmail.com>; Martijn Visser <ma...@apache.org>; dev@flink.apache.org <de...@flink.apache.org>; user <us...@flink.apache.org>
Subject: Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
EXTERNAL EMAIL
Thanks Chesnay for working on this. Would you like to share more info about the JDK bug?
Best regards,
Jing
On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>> wrote:
As it turns out Kryo isn't a blocker; we ran into a JDK bug.
On 31/03/2023 08:57, Chesnay Schepler wrote:
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
Kroy themselves state that v5 likely can't read v2 data.
However, both versions can be on the classpath without classpath as v5 offers a versioned artifact that includes the version in the package.
It probably wouldn't be difficult to migrate a savepoint to Kryo v5, purely from a read/write perspective.
The bigger question is how we expose this new Kryo version in the API. If we stick to the versioned jar we need to either duplicate all current Kryo-related APIs or find a better way to integrate other serialization stacks.
On 30/03/2023 17:50, Piotr Nowojski wrote:
Hey,
> 1. The Flink community agrees that we upgrade Kryo to a later version, which means breaking all checkpoint/savepoint compatibility and releasing a Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support dropped. This is probably the quickest way, but would still mean that we expose Kryo in the Flink APIs, which is the main reason why we haven't been able to upgrade Kryo at all.
This sounds pretty bad to me.
Has anyone looked into what it would take to provide a smooth migration from Kryo2 -> Kryo5?
Best,
Piotrek
czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>> napisał(a):
Hi Martijn,
just to be sure, if all state-related classes use a POJO serializer, Kryo will never come into play, right? Given FLINK-16686 [1], I wonder how many users actually have jobs with Kryo and RocksDB, but even if there aren't many, that still leaves those who don't use RocksDB for checkpoints/savepoints.
If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let users choose between v2/v5 jars by separating them like log4j2 jars?
[1] https://issues.apache.org/jira/browse/FLINK-16686
Regards,
Alexis.
Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <ma...@apache.org>>:
Hi all,
I also saw a thread on this topic from Clayton Wohl [1] on this topic, which I'm including in this discussion thread to avoid that it gets lost.
From my perspective, there's two main ways to get to Java 17:
1. The Flink community agrees that we upgrade Kryo to a later version, which means breaking all checkpoint/savepoint compatibility and releasing a Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support dropped. This is probably the quickest way, but would still mean that we expose Kryo in the Flink APIs, which is the main reason why we haven't been able to upgrade Kryo at all.
2. There's a contributor who makes a contribution that bumps Kryo, but either a) automagically reads in all old checkpoints/savepoints in using Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned in the Kryo migration guide [2][3] or b) provides an offline tool that allows users that are interested in migrating their snapshots manually before starting from a newer version. That potentially could prevent the need to introduce a new Flink major version. In both scenarios, ideally the contributor would also help with avoiding the exposure of Kryo so that we will be in a better shape in the future.
It would be good to get the opinion of the community for either of these two options, or potentially for another one that I haven't mentioned. If it appears that there's an overall agreement on the direction, I would propose that a FLIP gets created which describes the entire process.
Looking forward to the thoughts of others, including the Users (therefore including the User ML).
Best regards,
Martijn
[1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
[2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
[3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>> wrote:
I agree, there are several options to mitigate the migration from v2 to v5.
yet, Oracle roadmap is to end JDK 11 support in September this year.
________________________________
From: ConradJam <ja...@gmail.com>>
Sent: Thursday, March 16, 2023 4:36 AM
To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
Subject: Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
EXTERNAL EMAIL
Thanks for your start this discuss
I have been tracking this problem for a long time, until I saw a
conversation in ISSUSE a few days ago and learned that the Kryo version
problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
As @cherry said it ruined our whole effort towards JDK17
I am in favor of providing an external tool to migrate from Kryo old
version checkpoint to the new Kryo new checkpoint at one time (Maybe this
tool start in flink 2.0 ?), does this tool currently have any plans or
ideas worth discuss
I think it should not be difficult to be compatible with JDK11 and JDK17.
We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
that it is marked as Deprecated [2]
Here I add that we need to pay attention to the version of Scala and the
version of JDK17
[1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
https://issues.apache.org/jira/browse/FLINK-24998
[2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
Tamir Sagi <Ta...@niceactimize.com>> 于2023年3月16日周四 00:54写道：
> Hey dev community,
>
> I'm writing this email to kick off a discussion following this epic:
> FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>
> We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
> currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet, with
> no timeline, the reason, based on the aforementioned ticket is the
> following tickets
>
> 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> https://issues.apache.org/jira/browse/FLINK-24998>.
> 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> version<https://issues.apache.org/jira/browse/FLINK-3154>
>
> My question is whether it is possible to release a major version (Flink
> 2.0.0) using the latest Kryo version for those who don't need to restore
> old savepoints/checkpoints in newer format.
>
> 1. Leverage JDK 17 features within JVM
> 2. Moving from the old format to the newer one will be handled only
> once - a mitigation can be achieved by a conversion tool or external
> serializers, both can be provided later on.
>
> I'd like to emphasize that the next JDK LTS (21) will be released this
> September. furthermore, Flink already supports JDK 12-15, which is very
> close to JDK 17 (LTS) - that was released in September 2021. JDK 11 will
> become a legacy soon, as more frameworks moving towards JDK 17 and are less
> likely to support JDK 11 in the near future. (For example, Spring Boot 3
> requires JDK 17 already).
>
> Thank you for your consideration of my request.
>
> Tamir.
>
>
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
--
Best
ConradJam
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Tamir Sagi <Ta...@niceactimize.com>.
More details about the JDK bug here
https://bugs.openjdk.org/browse/JDK-8277529
Related Jira ticket
https://issues.apache.org/jira/browse/FLINK-24998
________________________________
From: Jing Ge via user <us...@flink.apache.org>
Sent: Monday, April 24, 2023 11:15 PM
To: Chesnay Schepler <ch...@apache.org>
Cc: Piotr Nowojski <pn...@apache.org>; Alexis Sarda-Espinosa <sa...@gmail.com>; Martijn Visser <ma...@apache.org>; dev@flink.apache.org <de...@flink.apache.org>; user <us...@flink.apache.org>
Subject: Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
EXTERNAL EMAIL
Thanks Chesnay for working on this. Would you like to share more info about the JDK bug?
Best regards,
Jing
On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>> wrote:
As it turns out Kryo isn't a blocker; we ran into a JDK bug.
On 31/03/2023 08:57, Chesnay Schepler wrote:
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
Kroy themselves state that v5 likely can't read v2 data.
However, both versions can be on the classpath without classpath as v5 offers a versioned artifact that includes the version in the package.
It probably wouldn't be difficult to migrate a savepoint to Kryo v5, purely from a read/write perspective.
The bigger question is how we expose this new Kryo version in the API. If we stick to the versioned jar we need to either duplicate all current Kryo-related APIs or find a better way to integrate other serialization stacks.
On 30/03/2023 17:50, Piotr Nowojski wrote:
Hey,
> 1. The Flink community agrees that we upgrade Kryo to a later version, which means breaking all checkpoint/savepoint compatibility and releasing a Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support dropped. This is probably the quickest way, but would still mean that we expose Kryo in the Flink APIs, which is the main reason why we haven't been able to upgrade Kryo at all.
This sounds pretty bad to me.
Has anyone looked into what it would take to provide a smooth migration from Kryo2 -> Kryo5?
Best,
Piotrek
czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>> napisał(a):
Hi Martijn,
just to be sure, if all state-related classes use a POJO serializer, Kryo will never come into play, right? Given FLINK-16686 [1], I wonder how many users actually have jobs with Kryo and RocksDB, but even if there aren't many, that still leaves those who don't use RocksDB for checkpoints/savepoints.
If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let users choose between v2/v5 jars by separating them like log4j2 jars?
[1] https://issues.apache.org/jira/browse/FLINK-16686
Regards,
Alexis.
Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <ma...@apache.org>>:
Hi all,
I also saw a thread on this topic from Clayton Wohl [1] on this topic, which I'm including in this discussion thread to avoid that it gets lost.
From my perspective, there's two main ways to get to Java 17:
1. The Flink community agrees that we upgrade Kryo to a later version, which means breaking all checkpoint/savepoint compatibility and releasing a Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support dropped. This is probably the quickest way, but would still mean that we expose Kryo in the Flink APIs, which is the main reason why we haven't been able to upgrade Kryo at all.
2. There's a contributor who makes a contribution that bumps Kryo, but either a) automagically reads in all old checkpoints/savepoints in using Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned in the Kryo migration guide [2][3] or b) provides an offline tool that allows users that are interested in migrating their snapshots manually before starting from a newer version. That potentially could prevent the need to introduce a new Flink major version. In both scenarios, ideally the contributor would also help with avoiding the exposure of Kryo so that we will be in a better shape in the future.
It would be good to get the opinion of the community for either of these two options, or potentially for another one that I haven't mentioned. If it appears that there's an overall agreement on the direction, I would propose that a FLIP gets created which describes the entire process.
Looking forward to the thoughts of others, including the Users (therefore including the User ML).
Best regards,
Martijn
[1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
[2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
[3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>> wrote:
I agree, there are several options to mitigate the migration from v2 to v5.
yet, Oracle roadmap is to end JDK 11 support in September this year.
________________________________
From: ConradJam <ja...@gmail.com>>
Sent: Thursday, March 16, 2023 4:36 AM
To: dev@flink.apache.org<ma...@flink.apache.org> <de...@flink.apache.org>>
Subject: Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
EXTERNAL EMAIL
Thanks for your start this discuss
I have been tracking this problem for a long time, until I saw a
conversation in ISSUSE a few days ago and learned that the Kryo version
problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
As @cherry said it ruined our whole effort towards JDK17
I am in favor of providing an external tool to migrate from Kryo old
version checkpoint to the new Kryo new checkpoint at one time (Maybe this
tool start in flink 2.0 ?), does this tool currently have any plans or
ideas worth discuss
I think it should not be difficult to be compatible with JDK11 and JDK17.
We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
that it is marked as Deprecated [2]
Here I add that we need to pay attention to the version of Scala and the
version of JDK17
[1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
https://issues.apache.org/jira/browse/FLINK-24998
[2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
Tamir Sagi <Ta...@niceactimize.com>> 于2023年3月16日周四 00:54写道：
> Hey dev community,
>
> I'm writing this email to kick off a discussion following this epic:
> FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>
> We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
> currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet, with
> no timeline, the reason, based on the aforementioned ticket is the
> following tickets
>
> 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
> https://issues.apache.org/jira/browse/FLINK-24998>.
> 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
> version<https://issues.apache.org/jira/browse/FLINK-3154>
>
> My question is whether it is possible to release a major version (Flink
> 2.0.0) using the latest Kryo version for those who don't need to restore
> old savepoints/checkpoints in newer format.
>
> 1. Leverage JDK 17 features within JVM
> 2. Moving from the old format to the newer one will be handled only
> once - a mitigation can be achieved by a conversion tool or external
> serializers, both can be provided later on.
>
> I'd like to emphasize that the next JDK LTS (21) will be released this
> September. furthermore, Flink already supports JDK 12-15, which is very
> close to JDK 17 (LTS) - that was released in September 2021. JDK 11 will
> become a legacy soon, as more frameworks moving towards JDK 17 and are less
> likely to support JDK 11 in the near future. (For example, Spring Boot 3
> requires JDK 17 already).
>
> Thank you for your consideration of my request.
>
> Tamir.
>
>
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
--
Best
ConradJam
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Jing Ge via user <us...@flink.apache.org>.
Thanks Chesnay for working on this. Would you like to share more info about
the JDK bug?
Best regards,
Jing
On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
wrote:
> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>
> On 31/03/2023 08:57, Chesnay Schepler wrote:
>
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API. If
> we stick to the versioned jar we need to either duplicate all current
> Kryo-related APIs or find a better way to integrate other serialization
> stacks.
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth migration
> from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
> napisał(a):
>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvisser@apache.org>:
>>
>>> Hi all,
>>>
>>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>>> which I'm including in this discussion thread to avoid that it gets lost.
>>>
>>> From my perspective, there's two main ways to get to Java 17:
>>>
>>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>>> which means breaking all checkpoint/savepoint compatibility and releasing a
>>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>>> dropped. This is probably the quickest way, but would still mean that we
>>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>>> able to upgrade Kryo at all.
>>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>>> either a) automagically reads in all old checkpoints/savepoints in using
>>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>>> allows users that are interested in migrating their snapshots manually
>>> before starting from a newer version. That potentially could prevent the
>>> need to introduce a new Flink major version. In both scenarios, ideally the
>>> contributor would also help with avoiding the exposure of Kryo so that we
>>> will be in a better shape in the future.
>>>
>>> It would be good to get the opinion of the community for either of these
>>> two options, or potentially for another one that I haven't mentioned. If it
>>> appears that there's an overall agreement on the direction, I would propose
>>> that a FLIP gets created which describes the entire process.
>>>
>>> Looking forward to the thoughts of others, including the Users
>>> (therefore including the User ML).
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>>
>>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
>>> wrote:
>>>
>>>> I agree, there are several options to mitigate the migration from v2 to
>>>> v5.
>>>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: ConradJam <ja...@gmail.com>
>>>> Sent: Thursday, March 16, 2023 4:36 AM
>>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>>> Subject: Re: [Discussion] - Release major Flink version to support JDK
>>>> 17 (LTS)
>>>>
>>>> EXTERNAL EMAIL
>>>>
>>>>
>>>>
>>>> Thanks for your start this discuss
>>>>
>>>>
>>>> I have been tracking this problem for a long time, until I saw a
>>>> conversation in ISSUSE a few days ago and learned that the Kryo version
>>>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>>>
>>>> As @cherry said it ruined our whole effort towards JDK17
>>>>
>>>> I am in favor of providing an external tool to migrate from Kryo old
>>>> version checkpoint to the new Kryo new checkpoint at one time (Maybe
>>>> this
>>>> tool start in flink 2.0 ?), does this tool currently have any plans or
>>>> ideas worth discuss
>>>>
>>>>
>>>> I think it should not be difficult to be compatible with JDK11 and
>>>> JDK17.
>>>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>>>> that it is marked as Deprecated [2]
>>>>
>>>>
>>>> Here I add that we need to pay attention to the version of Scala and the
>>>> version of JDK17
>>>>
>>>>
>>>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>>>> https://issues.apache.org/jira/browse/FLINK-24998
>>>>
>>>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
>>>> instead
>>>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>>>
>>>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>>>
>>>> > Hey dev community,
>>>> >
>>>> > I'm writing this email to kick off a discussion following this epic:
>>>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>>>> >
>>>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
>>>> which
>>>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>>>> with
>>>> > no timeline, the reason, based on the aforementioned ticket is the
>>>> > following tickets
>>>> >
>>>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>>>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>>>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>>>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>>>> >
>>>> > My question is whether it is possible to release a major version
>>>> (Flink
>>>> > 2.0.0) using the latest Kryo version for those who don't need to
>>>> restore
>>>> > old savepoints/checkpoints in newer format.
>>>> >
>>>> > 1. Leverage JDK 17 features within JVM
>>>> > 2. Moving from the old format to the newer one will be handled only
>>>> > once - a mitigation can be achieved by a conversion tool or external
>>>> > serializers, both can be provided later on.
>>>> >
>>>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>>>> > September. furthermore, Flink already supports JDK 12-15, which is
>>>> very
>>>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>>>> will
>>>> > become a legacy soon, as more frameworks moving towards JDK 17 and
>>>> are less
>>>> > likely to support JDK 11 in the near future. (For example, Spring
>>>> Boot 3
>>>> > requires JDK 17 already).
>>>> >
>>>> > Thank you for your consideration of my request.
>>>> >
>>>> > Tamir.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Confidentiality: This communication and any attachments are intended
>>>> for
>>>> > the above-named persons only and may be confidential and/or legally
>>>> > privileged. Any opinions expressed in this communication are not
>>>> > necessarily those of NICE Actimize. If this communication has come to
>>>> you
>>>> > in error you must take no action based on it, nor must you copy or
>>>> show it
>>>> > to anyone; please delete/destroy and inform the sender by e-mail
>>>> > immediately.
>>>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>>> > Viruses: Although we have taken steps toward ensuring that this
>>>> e-mail and
>>>> > attachments are free from any virus, we advise that in keeping with
>>>> good
>>>> > computing practice the recipient should ensure they are actually
>>>> virus free.
>>>> >
>>>>
>>>>
>>>> --
>>>> Best
>>>>
>>>> ConradJam
>>>>
>>>> Confidentiality: This communication and any attachments are intended
>>>> for the above-named persons only and may be confidential and/or legally
>>>> privileged. Any opinions expressed in this communication are not
>>>> necessarily those of NICE Actimize. If this communication has come to you
>>>> in error you must take no action based on it, nor must you copy or show it
>>>> to anyone; please delete/destroy and inform the sender by e-mail
>>>> immediately.
>>>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>>> Viruses: Although we have taken steps toward ensuring that this e-mail
>>>> and attachments are free from any virus, we advise that in keeping with
>>>> good computing practice the recipient should ensure they are actually virus
>>>> free.
>>>>
>>>
>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Thanks Chesnay for working on this. Would you like to share more info about
the JDK bug?
Best regards,
Jing
On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler <ch...@apache.org>
wrote:
> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>
> On 31/03/2023 08:57, Chesnay Schepler wrote:
>
>
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API. If
> we stick to the versioned jar we need to either duplicate all current
> Kryo-related APIs or find a better way to integrate other serialization
> stacks.
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth migration
> from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
> napisał(a):
>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
>> users choose between v2/v5 jars by separating them like log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvisser@apache.org>:
>>
>>> Hi all,
>>>
>>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>>> which I'm including in this discussion thread to avoid that it gets lost.
>>>
>>> From my perspective, there's two main ways to get to Java 17:
>>>
>>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>>> which means breaking all checkpoint/savepoint compatibility and releasing a
>>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>>> dropped. This is probably the quickest way, but would still mean that we
>>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>>> able to upgrade Kryo at all.
>>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>>> either a) automagically reads in all old checkpoints/savepoints in using
>>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>>> allows users that are interested in migrating their snapshots manually
>>> before starting from a newer version. That potentially could prevent the
>>> need to introduce a new Flink major version. In both scenarios, ideally the
>>> contributor would also help with avoiding the exposure of Kryo so that we
>>> will be in a better shape in the future.
>>>
>>> It would be good to get the opinion of the community for either of these
>>> two options, or potentially for another one that I haven't mentioned. If it
>>> appears that there's an overall agreement on the direction, I would propose
>>> that a FLIP gets created which describes the entire process.
>>>
>>> Looking forward to the thoughts of others, including the Users
>>> (therefore including the User ML).
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>>
>>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
>>> wrote:
>>>
>>>> I agree, there are several options to mitigate the migration from v2 to
>>>> v5.
>>>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: ConradJam <ja...@gmail.com>
>>>> Sent: Thursday, March 16, 2023 4:36 AM
>>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>>> Subject: Re: [Discussion] - Release major Flink version to support JDK
>>>> 17 (LTS)
>>>>
>>>> EXTERNAL EMAIL
>>>>
>>>>
>>>>
>>>> Thanks for your start this discuss
>>>>
>>>>
>>>> I have been tracking this problem for a long time, until I saw a
>>>> conversation in ISSUSE a few days ago and learned that the Kryo version
>>>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>>>
>>>> As @cherry said it ruined our whole effort towards JDK17
>>>>
>>>> I am in favor of providing an external tool to migrate from Kryo old
>>>> version checkpoint to the new Kryo new checkpoint at one time (Maybe
>>>> this
>>>> tool start in flink 2.0 ?), does this tool currently have any plans or
>>>> ideas worth discuss
>>>>
>>>>
>>>> I think it should not be difficult to be compatible with JDK11 and
>>>> JDK17.
>>>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>>>> that it is marked as Deprecated [2]
>>>>
>>>>
>>>> Here I add that we need to pay attention to the version of Scala and the
>>>> version of JDK17
>>>>
>>>>
>>>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>>>> https://issues.apache.org/jira/browse/FLINK-24998
>>>>
>>>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
>>>> instead
>>>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>>>
>>>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>>>
>>>> > Hey dev community,
>>>> >
>>>> > I'm writing this email to kick off a discussion following this epic:
>>>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>>>> >
>>>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
>>>> which
>>>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>>>> with
>>>> > no timeline, the reason, based on the aforementioned ticket is the
>>>> > following tickets
>>>> >
>>>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>>>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>>>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>>>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>>>> >
>>>> > My question is whether it is possible to release a major version
>>>> (Flink
>>>> > 2.0.0) using the latest Kryo version for those who don't need to
>>>> restore
>>>> > old savepoints/checkpoints in newer format.
>>>> >
>>>> > 1. Leverage JDK 17 features within JVM
>>>> > 2. Moving from the old format to the newer one will be handled only
>>>> > once - a mitigation can be achieved by a conversion tool or external
>>>> > serializers, both can be provided later on.
>>>> >
>>>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>>>> > September. furthermore, Flink already supports JDK 12-15, which is
>>>> very
>>>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>>>> will
>>>> > become a legacy soon, as more frameworks moving towards JDK 17 and
>>>> are less
>>>> > likely to support JDK 11 in the near future. (For example, Spring
>>>> Boot 3
>>>> > requires JDK 17 already).
>>>> >
>>>> > Thank you for your consideration of my request.
>>>> >
>>>> > Tamir.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Confidentiality: This communication and any attachments are intended
>>>> for
>>>> > the above-named persons only and may be confidential and/or legally
>>>> > privileged. Any opinions expressed in this communication are not
>>>> > necessarily those of NICE Actimize. If this communication has come to
>>>> you
>>>> > in error you must take no action based on it, nor must you copy or
>>>> show it
>>>> > to anyone; please delete/destroy and inform the sender by e-mail
>>>> > immediately.
>>>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>>> > Viruses: Although we have taken steps toward ensuring that this
>>>> e-mail and
>>>> > attachments are free from any virus, we advise that in keeping with
>>>> good
>>>> > computing practice the recipient should ensure they are actually
>>>> virus free.
>>>> >
>>>>
>>>>
>>>> --
>>>> Best
>>>>
>>>> ConradJam
>>>>
>>>> Confidentiality: This communication and any attachments are intended
>>>> for the above-named persons only and may be confidential and/or legally
>>>> privileged. Any opinions expressed in this communication are not
>>>> necessarily those of NICE Actimize. If this communication has come to you
>>>> in error you must take no action based on it, nor must you copy or show it
>>>> to anyone; please delete/destroy and inform the sender by e-mail
>>>> immediately.
>>>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>>> Viruses: Although we have taken steps toward ensuring that this e-mail
>>>> and attachments are free from any virus, we advise that in keeping with
>>>> good computing practice the recipient should ensure they are actually virus
>>>> free.
>>>>
>>>
>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Chesnay Schepler <ch...@apache.org>.
As it turns out Kryo isn't a blocker; we ran into a JDK bug.
On 31/03/2023 08:57, Chesnay Schepler wrote:
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API.
> If we stick to the versioned jar we need to either duplicate all
> current Kryo-related APIs or find a better way to integrate other
> serialization stacks.
>
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later
>> version, which means breaking all checkpoint/savepoint compatibility
>> and releasing a Flink 2.0 with Java 17 support added and Java 8 and
>> Flink Scala API support dropped. This is probably the quickest way,
>> but would still mean that we expose Kryo in the Flink APIs, which is
>> the main reason why we haven't been able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth
>> migration from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
>> <sa...@gmail.com> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO
>> serializer, Kryo will never come into play, right? Given
>> FLINK-16686 [1], I wonder how many users actually have jobs with
>> Kryo and RocksDB, but even if there aren't many, that still
>> leaves those who don't use RocksDB for checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible
>> to let users choose between v2/v5 jars by separating them like
>> log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser
>> <ma...@apache.org>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on
>> this topic, which I'm including in this discussion thread to
>> avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later
>> version, which means breaking all checkpoint/savepoint
>> compatibility and releasing a Flink 2.0 with Java 17 support
>> added and Java 8 and Flink Scala API support dropped. This is
>> probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why
>> we haven't been able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps
>> Kryo, but either a) automagically reads in all old
>> checkpoints/savepoints in using Kryo v2 and writes them to
>> new snapshots using Kryo v5 (like is mentioned in the Kryo
>> migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots
>> manually before starting from a newer version. That
>> potentially could prevent the need to introduce a new Flink
>> major version. In both scenarios, ideally the contributor
>> would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for
>> either of these two options, or potentially for another one
>> that I haven't mentioned. If it appears that there's an
>> overall agreement on the direction, I would propose that a
>> FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the
>> Users (therefore including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1]
>> https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2]
>> https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
>> <Ta...@niceactimize.com> wrote:
>>
>> I agree, there are several options to mitigate the
>> migration from v2 to v5.
>> yet, Oracle roadmap is to end JDK 11 support in September
>> this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [Discussion] - Release major Flink version
>> to support JDK 17 (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until
>> I saw a
>> conversation in ISSUSE a few days ago and learned that
>> the Kryo version
>> problem will affect the JDK17 compilation of snapshots
>> [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate
>> from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one
>> time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have
>> any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with
>> JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also
>> mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version
>> of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on
>> Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to
>> deprecate Java 8 instead
>> of requiring Java 11
>> https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四
>> 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion
>> following this epic:
>> >
>> FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker
>> now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not
>> support JDK 17 yet, with
>> > no timeline, the reason, based on the aforementioned
>> ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread
>> on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to
>> latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a
>> major version (Flink
>> > 2.0.0) using the latest Kryo version for those who
>> don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will
>> be handled only
>> > once - a mitigation can be achieved by a conversion
>> tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will
>> be released this
>> > September. furthermore, Flink already supports JDK
>> 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September
>> 2021. JDK 11 will
>> > become a legacy soon, as more frameworks moving towards
>> JDK 17 and are less
>> > likely to support JDK 11 in the near future. (For
>> example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments
>> are intended for
>> > the above-named persons only and may be confidential
>> and/or legally
>> > privileged. Any opinions expressed in this
>> communication are not
>> > necessarily those of NICE Actimize. If this
>> communication has come to you
>> > in error you must take no action based on it, nor must
>> you copy or show it
>> > to anyone; please delete/destroy and inform the sender
>> by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and
>> outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring
>> that this e-mail and
>> > attachments are free from any virus, we advise that in
>> keeping with good
>> > computing practice the recipient should ensure they are
>> actually virus free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments
>> are intended for the above-named persons only and may be
>> confidential and/or legally privileged. Any opinions
>> expressed in this communication are not necessarily those
>> of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must
>> you copy or show it to anyone; please delete/destroy and
>> inform the sender by e-mail immediately.
>> Monitoring: NICE Actimize may monitor incoming and
>> outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring
>> that this e-mail and attachments are free from any virus,
>> we advise that in keeping with good computing practice
>> the recipient should ensure they are actually virus free.
>>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Chesnay Schepler <ch...@apache.org>.
As it turns out Kryo isn't a blocker; we ran into a JDK bug.
On 31/03/2023 08:57, Chesnay Schepler wrote:
> https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>
> Kroy themselves state that v5 likely can't read v2 data.
>
> However, both versions can be on the classpath without classpath as v5
> offers a versioned artifact that includes the version in the package.
>
> It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
> purely from a read/write perspective.
>
> The bigger question is how we expose this new Kryo version in the API.
> If we stick to the versioned jar we need to either duplicate all
> current Kryo-related APIs or find a better way to integrate other
> serialization stacks.
>
> On 30/03/2023 17:50, Piotr Nowojski wrote:
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later
>> version, which means breaking all checkpoint/savepoint compatibility
>> and releasing a Flink 2.0 with Java 17 support added and Java 8 and
>> Flink Scala API support dropped. This is probably the quickest way,
>> but would still mean that we expose Kryo in the Flink APIs, which is
>> the main reason why we haven't been able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth
>> migration from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
>> <sa...@gmail.com> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO
>> serializer, Kryo will never come into play, right? Given
>> FLINK-16686 [1], I wonder how many users actually have jobs with
>> Kryo and RocksDB, but even if there aren't many, that still
>> leaves those who don't use RocksDB for checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it impossible
>> to let users choose between v2/v5 jars by separating them like
>> log4j2 jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser
>> <ma...@apache.org>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on
>> this topic, which I'm including in this discussion thread to
>> avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later
>> version, which means breaking all checkpoint/savepoint
>> compatibility and releasing a Flink 2.0 with Java 17 support
>> added and Java 8 and Flink Scala API support dropped. This is
>> probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why
>> we haven't been able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps
>> Kryo, but either a) automagically reads in all old
>> checkpoints/savepoints in using Kryo v2 and writes them to
>> new snapshots using Kryo v5 (like is mentioned in the Kryo
>> migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots
>> manually before starting from a newer version. That
>> potentially could prevent the need to introduce a new Flink
>> major version. In both scenarios, ideally the contributor
>> would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for
>> either of these two options, or potentially for another one
>> that I haven't mentioned. If it appears that there's an
>> overall agreement on the direction, I would propose that a
>> FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the
>> Users (therefore including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1]
>> https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2]
>> https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
>> <Ta...@niceactimize.com> wrote:
>>
>> I agree, there are several options to mitigate the
>> migration from v2 to v5.
>> yet, Oracle roadmap is to end JDK 11 support in September
>> this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [Discussion] - Release major Flink version
>> to support JDK 17 (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until
>> I saw a
>> conversation in ISSUSE a few days ago and learned that
>> the Kryo version
>> problem will affect the JDK17 compilation of snapshots
>> [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate
>> from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one
>> time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have
>> any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with
>> JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also
>> mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version
>> of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on
>> Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to
>> deprecate Java 8 instead
>> of requiring Java 11
>> https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四
>> 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion
>> following this epic:
>> >
>> FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker
>> now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not
>> support JDK 17 yet, with
>> > no timeline, the reason, based on the aforementioned
>> ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread
>> on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to
>> latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a
>> major version (Flink
>> > 2.0.0) using the latest Kryo version for those who
>> don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will
>> be handled only
>> > once - a mitigation can be achieved by a conversion
>> tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will
>> be released this
>> > September. furthermore, Flink already supports JDK
>> 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September
>> 2021. JDK 11 will
>> > become a legacy soon, as more frameworks moving towards
>> JDK 17 and are less
>> > likely to support JDK 11 in the near future. (For
>> example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments
>> are intended for
>> > the above-named persons only and may be confidential
>> and/or legally
>> > privileged. Any opinions expressed in this
>> communication are not
>> > necessarily those of NICE Actimize. If this
>> communication has come to you
>> > in error you must take no action based on it, nor must
>> you copy or show it
>> > to anyone; please delete/destroy and inform the sender
>> by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and
>> outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring
>> that this e-mail and
>> > attachments are free from any virus, we advise that in
>> keeping with good
>> > computing practice the recipient should ensure they are
>> actually virus free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments
>> are intended for the above-named persons only and may be
>> confidential and/or legally privileged. Any opinions
>> expressed in this communication are not necessarily those
>> of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must
>> you copy or show it to anyone; please delete/destroy and
>> inform the sender by e-mail immediately.
>> Monitoring: NICE Actimize may monitor incoming and
>> outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring
>> that this e-mail and attachments are free from any virus,
>> we advise that in keeping with good computing practice
>> the recipient should ensure they are actually virus free.
>>
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Chesnay Schepler <ch...@apache.org>.
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
Kroy themselves state that v5 likely can't read v2 data.
However, both versions can be on the classpath without classpath as v5
offers a versioned artifact that includes the version in the package.
It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
purely from a read/write perspective.
The bigger question is how we expose this new Kryo version in the API.
If we stick to the versioned jar we need to either duplicate all current
Kryo-related APIs or find a better way to integrate other serialization
stacks.
On 30/03/2023 17:50, Piotr Nowojski wrote:
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later
> version, which means breaking all checkpoint/savepoint compatibility
> and releasing a Flink 2.0 with Java 17 support added and Java 8 and
> Flink Scala API support dropped. This is probably the quickest way,
> but would still mean that we expose Kryo in the Flink APIs, which is
> the main reason why we haven't been able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth
> migration from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
> <sa...@gmail.com> napisał(a):
>
> Hi Martijn,
>
> just to be sure, if all state-related classes use a POJO
> serializer, Kryo will never come into play, right? Given
> FLINK-16686 [1], I wonder how many users actually have jobs with
> Kryo and RocksDB, but even if there aren't many, that still leaves
> those who don't use RocksDB for checkpoints/savepoints.
>
> If Kryo were to stay in the Flink APIs in v1.X, is it impossible
> to let users choose between v2/v5 jars by separating them like
> log4j2 jars?
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
> Regards,
> Alexis.
>
> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser
> <ma...@apache.org>:
>
> Hi all,
>
> I also saw a thread on this topic from Clayton Wohl [1] on
> this topic, which I'm including in this discussion thread to
> avoid that it gets lost.
>
> From my perspective, there's two main ways to get to Java 17:
>
> 1. The Flink community agrees that we upgrade Kryo to a later
> version, which means breaking all checkpoint/savepoint
> compatibility and releasing a Flink 2.0 with Java 17 support
> added and Java 8 and Flink Scala API support dropped. This is
> probably the quickest way, but would still mean that we expose
> Kryo in the Flink APIs, which is the main reason why we
> haven't been able to upgrade Kryo at all.
> 2. There's a contributor who makes a contribution that bumps
> Kryo, but either a) automagically reads in all old
> checkpoints/savepoints in using Kryo v2 and writes them to new
> snapshots using Kryo v5 (like is mentioned in the Kryo
> migration guide [2][3] or b) provides an offline tool that
> allows users that are interested in migrating their snapshots
> manually before starting from a newer version. That
> potentially could prevent the need to introduce a new Flink
> major version. In both scenarios, ideally the contributor
> would also help with avoiding the exposure of Kryo so that we
> will be in a better shape in the future.
>
> It would be good to get the opinion of the community for
> either of these two options, or potentially for another one
> that I haven't mentioned. If it appears that there's an
> overall agreement on the direction, I would propose that a
> FLIP gets created which describes the entire process.
>
> Looking forward to the thoughts of others, including the Users
> (therefore including the User ML).
>
> Best regards,
>
> Martijn
>
> [1]
> https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> [2]
> https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>
> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
> <Ta...@niceactimize.com> wrote:
>
> I agree, there are several options to mitigate the
> migration from v2 to v5.
> yet, Oracle roadmap is to end JDK 11 support in September
> this year.
>
>
>
> ________________________________
> From: ConradJam <ja...@gmail.com>
> Sent: Thursday, March 16, 2023 4:36 AM
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [Discussion] - Release major Flink version to
> support JDK 17 (LTS)
>
> EXTERNAL EMAIL
>
>
>
> Thanks for your start this discuss
>
>
> I have been tracking this problem for a long time, until I
> saw a
> conversation in ISSUSE a few days ago and learned that the
> Kryo version
> problem will affect the JDK17 compilation of snapshots [1]
> FLINK-24998 ,
>
> As @cherry said it ruined our whole effort towards JDK17
>
> I am in favor of providing an external tool to migrate
> from Kryo old
> version checkpoint to the new Kryo new checkpoint at one
> time (Maybe this
> tool start in flink 2.0 ?), does this tool currently have
> any plans or
> ideas worth discuss
>
>
> I think it should not be difficult to be compatible with
> JDK11 and JDK17.
> We should indeed abandon JDK8 in 2.0.0. It is also
> mentioned in the doc
> that it is marked as Deprecated [2]
>
>
> Here I add that we need to pay attention to the version of
> Scala and the
> version of JDK17
>
>
> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> https://issues.apache.org/jira/browse/FLINK-24998
>
> [2] FLINK-30501 Update Flink build instruction to
> deprecate Java 8 instead
> of requiring Java 11
> https://issues.apache.org/jira/browse/FLINK-30501
>
> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四
> 00:54写道：
>
> > Hey dev community,
> >
> > I'm writing this email to kick off a discussion
> following this epic:
> >
> FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >
> > We are moving towards JDK 17 (LTS) , the only blocker
> now is Flink which
> > currently remains on JDK 11 (LTS). Flink does not
> support JDK 17 yet, with
> > no timeline, the reason, based on the aforementioned
> ticket is the
> > following tickets
> >
> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread
> on Java 17<
> > https://issues.apache.org/jira/browse/FLINK-24998>.
> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to
> latest Kryo LTS
> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >
> > My question is whether it is possible to release a major
> version (Flink
> > 2.0.0) using the latest Kryo version for those who don't
> need to restore
> > old savepoints/checkpoints in newer format.
> >
> > 1. Leverage JDK 17 features within JVM
> > 2. Moving from the old format to the newer one will
> be handled only
> > once - a mitigation can be achieved by a conversion tool
> or external
> > serializers, both can be provided later on.
> >
> > I'd like to emphasize that the next JDK LTS (21) will be
> released this
> > September. furthermore, Flink already supports JDK
> 12-15, which is very
> > close to JDK 17 (LTS) - that was released in September
> 2021. JDK 11 will
> > become a legacy soon, as more frameworks moving towards
> JDK 17 and are less
> > likely to support JDK 11 in the near future. (For
> example, Spring Boot 3
> > requires JDK 17 already).
> >
> > Thank you for your consideration of my request.
> >
> > Tamir.
> >
> >
> >
> >
> > Confidentiality: This communication and any attachments
> are intended for
> > the above-named persons only and may be confidential
> and/or legally
> > privileged. Any opinions expressed in this communication
> are not
> > necessarily those of NICE Actimize. If this
> communication has come to you
> > in error you must take no action based on it, nor must
> you copy or show it
> > to anyone; please delete/destroy and inform the sender
> by e-mail
> > immediately.
> > Monitoring: NICE Actimize may monitor incoming and
> outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring
> that this e-mail and
> > attachments are free from any virus, we advise that in
> keeping with good
> > computing practice the recipient should ensure they are
> actually virus free.
> >
>
>
> --
> Best
>
> ConradJam
>
> Confidentiality: This communication and any attachments
> are intended for the above-named persons only and may be
> confidential and/or legally privileged. Any opinions
> expressed in this communication are not necessarily those
> of NICE Actimize. If this communication has come to you in
> error you must take no action based on it, nor must you
> copy or show it to anyone; please delete/destroy and
> inform the sender by e-mail immediately.
> Monitoring: NICE Actimize may monitor incoming and
> outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that
> this e-mail and attachments are free from any virus, we
> advise that in keeping with good computing practice the
> recipient should ensure they are actually virus free.
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Chesnay Schepler <ch...@apache.org>.
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
Kroy themselves state that v5 likely can't read v2 data.
However, both versions can be on the classpath without classpath as v5
offers a versioned artifact that includes the version in the package.
It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
purely from a read/write perspective.
The bigger question is how we expose this new Kryo version in the API.
If we stick to the versioned jar we need to either duplicate all current
Kryo-related APIs or find a better way to integrate other serialization
stacks.
On 30/03/2023 17:50, Piotr Nowojski wrote:
> Hey,
>
> > 1. The Flink community agrees that we upgrade Kryo to a later
> version, which means breaking all checkpoint/savepoint compatibility
> and releasing a Flink 2.0 with Java 17 support added and Java 8 and
> Flink Scala API support dropped. This is probably the quickest way,
> but would still mean that we expose Kryo in the Flink APIs, which is
> the main reason why we haven't been able to upgrade Kryo at all.
>
> This sounds pretty bad to me.
>
> Has anyone looked into what it would take to provide a smooth
> migration from Kryo2 -> Kryo5?
>
> Best,
> Piotrek
>
> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
> <sa...@gmail.com> napisał(a):
>
> Hi Martijn,
>
> just to be sure, if all state-related classes use a POJO
> serializer, Kryo will never come into play, right? Given
> FLINK-16686 [1], I wonder how many users actually have jobs with
> Kryo and RocksDB, but even if there aren't many, that still leaves
> those who don't use RocksDB for checkpoints/savepoints.
>
> If Kryo were to stay in the Flink APIs in v1.X, is it impossible
> to let users choose between v2/v5 jars by separating them like
> log4j2 jars?
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
> Regards,
> Alexis.
>
> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser
> <ma...@apache.org>:
>
> Hi all,
>
> I also saw a thread on this topic from Clayton Wohl [1] on
> this topic, which I'm including in this discussion thread to
> avoid that it gets lost.
>
> From my perspective, there's two main ways to get to Java 17:
>
> 1. The Flink community agrees that we upgrade Kryo to a later
> version, which means breaking all checkpoint/savepoint
> compatibility and releasing a Flink 2.0 with Java 17 support
> added and Java 8 and Flink Scala API support dropped. This is
> probably the quickest way, but would still mean that we expose
> Kryo in the Flink APIs, which is the main reason why we
> haven't been able to upgrade Kryo at all.
> 2. There's a contributor who makes a contribution that bumps
> Kryo, but either a) automagically reads in all old
> checkpoints/savepoints in using Kryo v2 and writes them to new
> snapshots using Kryo v5 (like is mentioned in the Kryo
> migration guide [2][3] or b) provides an offline tool that
> allows users that are interested in migrating their snapshots
> manually before starting from a newer version. That
> potentially could prevent the need to introduce a new Flink
> major version. In both scenarios, ideally the contributor
> would also help with avoiding the exposure of Kryo so that we
> will be in a better shape in the future.
>
> It would be good to get the opinion of the community for
> either of these two options, or potentially for another one
> that I haven't mentioned. If it appears that there's an
> overall agreement on the direction, I would propose that a
> FLIP gets created which describes the entire process.
>
> Looking forward to the thoughts of others, including the Users
> (therefore including the User ML).
>
> Best regards,
>
> Martijn
>
> [1]
> https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> [2]
> https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>
> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
> <Ta...@niceactimize.com> wrote:
>
> I agree, there are several options to mitigate the
> migration from v2 to v5.
> yet, Oracle roadmap is to end JDK 11 support in September
> this year.
>
>
>
> ________________________________
> From: ConradJam <ja...@gmail.com>
> Sent: Thursday, March 16, 2023 4:36 AM
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [Discussion] - Release major Flink version to
> support JDK 17 (LTS)
>
> EXTERNAL EMAIL
>
>
>
> Thanks for your start this discuss
>
>
> I have been tracking this problem for a long time, until I
> saw a
> conversation in ISSUSE a few days ago and learned that the
> Kryo version
> problem will affect the JDK17 compilation of snapshots [1]
> FLINK-24998 ,
>
> As @cherry said it ruined our whole effort towards JDK17
>
> I am in favor of providing an external tool to migrate
> from Kryo old
> version checkpoint to the new Kryo new checkpoint at one
> time (Maybe this
> tool start in flink 2.0 ?), does this tool currently have
> any plans or
> ideas worth discuss
>
>
> I think it should not be difficult to be compatible with
> JDK11 and JDK17.
> We should indeed abandon JDK8 in 2.0.0. It is also
> mentioned in the doc
> that it is marked as Deprecated [2]
>
>
> Here I add that we need to pay attention to the version of
> Scala and the
> version of JDK17
>
>
> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
> https://issues.apache.org/jira/browse/FLINK-24998
>
> [2] FLINK-30501 Update Flink build instruction to
> deprecate Java 8 instead
> of requiring Java 11
> https://issues.apache.org/jira/browse/FLINK-30501
>
> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四
> 00:54写道：
>
> > Hey dev community,
> >
> > I'm writing this email to kick off a discussion
> following this epic:
> >
> FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
> >
> > We are moving towards JDK 17 (LTS) , the only blocker
> now is Flink which
> > currently remains on JDK 11 (LTS). Flink does not
> support JDK 17 yet, with
> > no timeline, the reason, based on the aforementioned
> ticket is the
> > following tickets
> >
> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread
> on Java 17<
> > https://issues.apache.org/jira/browse/FLINK-24998>.
> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to
> latest Kryo LTS
> > version<https://issues.apache.org/jira/browse/FLINK-3154>
> >
> > My question is whether it is possible to release a major
> version (Flink
> > 2.0.0) using the latest Kryo version for those who don't
> need to restore
> > old savepoints/checkpoints in newer format.
> >
> > 1. Leverage JDK 17 features within JVM
> > 2. Moving from the old format to the newer one will
> be handled only
> > once - a mitigation can be achieved by a conversion tool
> or external
> > serializers, both can be provided later on.
> >
> > I'd like to emphasize that the next JDK LTS (21) will be
> released this
> > September. furthermore, Flink already supports JDK
> 12-15, which is very
> > close to JDK 17 (LTS) - that was released in September
> 2021. JDK 11 will
> > become a legacy soon, as more frameworks moving towards
> JDK 17 and are less
> > likely to support JDK 11 in the near future. (For
> example, Spring Boot 3
> > requires JDK 17 already).
> >
> > Thank you for your consideration of my request.
> >
> > Tamir.
> >
> >
> >
> >
> > Confidentiality: This communication and any attachments
> are intended for
> > the above-named persons only and may be confidential
> and/or legally
> > privileged. Any opinions expressed in this communication
> are not
> > necessarily those of NICE Actimize. If this
> communication has come to you
> > in error you must take no action based on it, nor must
> you copy or show it
> > to anyone; please delete/destroy and inform the sender
> by e-mail
> > immediately.
> > Monitoring: NICE Actimize may monitor incoming and
> outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring
> that this e-mail and
> > attachments are free from any virus, we advise that in
> keeping with good
> > computing practice the recipient should ensure they are
> actually virus free.
> >
>
>
> --
> Best
>
> ConradJam
>
> Confidentiality: This communication and any attachments
> are intended for the above-named persons only and may be
> confidential and/or legally privileged. Any opinions
> expressed in this communication are not necessarily those
> of NICE Actimize. If this communication has come to you in
> error you must take no action based on it, nor must you
> copy or show it to anyone; please delete/destroy and
> inform the sender by e-mail immediately.
> Monitoring: NICE Actimize may monitor incoming and
> outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that
> this e-mail and attachments are free from any virus, we
> advise that in keeping with good computing practice the
> recipient should ensure they are actually virus free.
>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Piotr Nowojski <pn...@apache.org>.
Hey,
> 1. The Flink community agrees that we upgrade Kryo to a later version,
which means breaking all checkpoint/savepoint compatibility and releasing a
Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
dropped. This is probably the quickest way, but would still mean that we
expose Kryo in the Flink APIs, which is the main reason why we haven't been
able to upgrade Kryo at all.
This sounds pretty bad to me.
Has anyone looked into what it would take to provide a smooth migration
from Kryo2 -> Kryo5?
Best,
Piotrek
czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
napisał(a):
> Hi Martijn,
>
> just to be sure, if all state-related classes use a POJO serializer, Kryo
> will never come into play, right? Given FLINK-16686 [1], I wonder how many
> users actually have jobs with Kryo and RocksDB, but even if there aren't
> many, that still leaves those who don't use RocksDB for
> checkpoints/savepoints.
>
> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
> users choose between v2/v5 jars by separating them like log4j2 jars?
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
> Regards,
> Alexis.
>
> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> martijnvisser@apache.org>:
>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> which I'm including in this discussion thread to avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots manually
>> before starting from a newer version. That potentially could prevent the
>> need to introduce a new Flink major version. In both scenarios, ideally the
>> contributor would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either of these
>> two options, or potentially for another one that I haven't mentioned. If it
>> appears that there's an overall agreement on the direction, I would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users (therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
>> wrote:
>>
>>> I agree, there are several options to mitigate the migration from v2 to
>>> v5.
>>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>>
>>>
>>>
>>> ________________________________
>>> From: ConradJam <ja...@gmail.com>
>>> Sent: Thursday, March 16, 2023 4:36 AM
>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>> Subject: Re: [Discussion] - Release major Flink version to support JDK
>>> 17 (LTS)
>>>
>>> EXTERNAL EMAIL
>>>
>>>
>>>
>>> Thanks for your start this discuss
>>>
>>>
>>> I have been tracking this problem for a long time, until I saw a
>>> conversation in ISSUSE a few days ago and learned that the Kryo version
>>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>>
>>> As @cherry said it ruined our whole effort towards JDK17
>>>
>>> I am in favor of providing an external tool to migrate from Kryo old
>>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>>> tool start in flink 2.0 ?), does this tool currently have any plans or
>>> ideas worth discuss
>>>
>>>
>>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>>> that it is marked as Deprecated [2]
>>>
>>>
>>> Here I add that we need to pay attention to the version of Scala and the
>>> version of JDK17
>>>
>>>
>>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>>> https://issues.apache.org/jira/browse/FLINK-24998
>>>
>>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
>>> instead
>>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>>
>>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>>
>>> > Hey dev community,
>>> >
>>> > I'm writing this email to kick off a discussion following this epic:
>>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>>> >
>>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
>>> which
>>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>>> with
>>> > no timeline, the reason, based on the aforementioned ticket is the
>>> > following tickets
>>> >
>>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>>> >
>>> > My question is whether it is possible to release a major version (Flink
>>> > 2.0.0) using the latest Kryo version for those who don't need to
>>> restore
>>> > old savepoints/checkpoints in newer format.
>>> >
>>> > 1. Leverage JDK 17 features within JVM
>>> > 2. Moving from the old format to the newer one will be handled only
>>> > once - a mitigation can be achieved by a conversion tool or external
>>> > serializers, both can be provided later on.
>>> >
>>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>>> > September. furthermore, Flink already supports JDK 12-15, which is
>>> very
>>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>>> will
>>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>>> less
>>> > likely to support JDK 11 in the near future. (For example, Spring Boot
>>> 3
>>> > requires JDK 17 already).
>>> >
>>> > Thank you for your consideration of my request.
>>> >
>>> > Tamir.
>>> >
>>> >
>>> >
>>> >
>>> > Confidentiality: This communication and any attachments are intended
>>> for
>>> > the above-named persons only and may be confidential and/or legally
>>> > privileged. Any opinions expressed in this communication are not
>>> > necessarily those of NICE Actimize. If this communication has come to
>>> you
>>> > in error you must take no action based on it, nor must you copy or
>>> show it
>>> > to anyone; please delete/destroy and inform the sender by e-mail
>>> > immediately.
>>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>>> and
>>> > attachments are free from any virus, we advise that in keeping with
>>> good
>>> > computing practice the recipient should ensure they are actually virus
>>> free.
>>> >
>>>
>>>
>>> --
>>> Best
>>>
>>> ConradJam
>>>
>>> Confidentiality: This communication and any attachments are intended for
>>> the above-named persons only and may be confidential and/or legally
>>> privileged. Any opinions expressed in this communication are not
>>> necessarily those of NICE Actimize. If this communication has come to you
>>> in error you must take no action based on it, nor must you copy or show it
>>> to anyone; please delete/destroy and inform the sender by e-mail
>>> immediately.
>>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>> Viruses: Although we have taken steps toward ensuring that this e-mail
>>> and attachments are free from any virus, we advise that in keeping with
>>> good computing practice the recipient should ensure they are actually virus
>>> free.
>>>
>>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Piotr Nowojski <pn...@apache.org>.
Hey,
> 1. The Flink community agrees that we upgrade Kryo to a later version,
which means breaking all checkpoint/savepoint compatibility and releasing a
Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
dropped. This is probably the quickest way, but would still mean that we
expose Kryo in the Flink APIs, which is the main reason why we haven't been
able to upgrade Kryo at all.
This sounds pretty bad to me.
Has anyone looked into what it would take to provide a smooth migration
from Kryo2 -> Kryo5?
Best,
Piotrek
czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa <sa...@gmail.com>
napisał(a):
> Hi Martijn,
>
> just to be sure, if all state-related classes use a POJO serializer, Kryo
> will never come into play, right? Given FLINK-16686 [1], I wonder how many
> users actually have jobs with Kryo and RocksDB, but even if there aren't
> many, that still leaves those who don't use RocksDB for
> checkpoints/savepoints.
>
> If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
> users choose between v2/v5 jars by separating them like log4j2 jars?
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
> Regards,
> Alexis.
>
> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
> martijnvisser@apache.org>:
>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
>> which I'm including in this discussion thread to avoid that it gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later version,
>> which means breaking all checkpoint/savepoint compatibility and releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
>> dropped. This is probably the quickest way, but would still mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline tool that
>> allows users that are interested in migrating their snapshots manually
>> before starting from a newer version. That potentially could prevent the
>> need to introduce a new Flink major version. In both scenarios, ideally the
>> contributor would also help with avoiding the exposure of Kryo so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either of these
>> two options, or potentially for another one that I haven't mentioned. If it
>> appears that there's an overall agreement on the direction, I would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users (therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
>> wrote:
>>
>>> I agree, there are several options to mitigate the migration from v2 to
>>> v5.
>>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>>
>>>
>>>
>>> ________________________________
>>> From: ConradJam <ja...@gmail.com>
>>> Sent: Thursday, March 16, 2023 4:36 AM
>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>> Subject: Re: [Discussion] - Release major Flink version to support JDK
>>> 17 (LTS)
>>>
>>> EXTERNAL EMAIL
>>>
>>>
>>>
>>> Thanks for your start this discuss
>>>
>>>
>>> I have been tracking this problem for a long time, until I saw a
>>> conversation in ISSUSE a few days ago and learned that the Kryo version
>>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>>
>>> As @cherry said it ruined our whole effort towards JDK17
>>>
>>> I am in favor of providing an external tool to migrate from Kryo old
>>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>>> tool start in flink 2.0 ?), does this tool currently have any plans or
>>> ideas worth discuss
>>>
>>>
>>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>>> that it is marked as Deprecated [2]
>>>
>>>
>>> Here I add that we need to pay attention to the version of Scala and the
>>> version of JDK17
>>>
>>>
>>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>>> https://issues.apache.org/jira/browse/FLINK-24998
>>>
>>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8
>>> instead
>>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>>
>>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>>
>>> > Hey dev community,
>>> >
>>> > I'm writing this email to kick off a discussion following this epic:
>>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>>> >
>>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink
>>> which
>>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>>> with
>>> > no timeline, the reason, based on the aforementioned ticket is the
>>> > following tickets
>>> >
>>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>>> >
>>> > My question is whether it is possible to release a major version (Flink
>>> > 2.0.0) using the latest Kryo version for those who don't need to
>>> restore
>>> > old savepoints/checkpoints in newer format.
>>> >
>>> > 1. Leverage JDK 17 features within JVM
>>> > 2. Moving from the old format to the newer one will be handled only
>>> > once - a mitigation can be achieved by a conversion tool or external
>>> > serializers, both can be provided later on.
>>> >
>>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>>> > September. furthermore, Flink already supports JDK 12-15, which is
>>> very
>>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>>> will
>>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>>> less
>>> > likely to support JDK 11 in the near future. (For example, Spring Boot
>>> 3
>>> > requires JDK 17 already).
>>> >
>>> > Thank you for your consideration of my request.
>>> >
>>> > Tamir.
>>> >
>>> >
>>> >
>>> >
>>> > Confidentiality: This communication and any attachments are intended
>>> for
>>> > the above-named persons only and may be confidential and/or legally
>>> > privileged. Any opinions expressed in this communication are not
>>> > necessarily those of NICE Actimize. If this communication has come to
>>> you
>>> > in error you must take no action based on it, nor must you copy or
>>> show it
>>> > to anyone; please delete/destroy and inform the sender by e-mail
>>> > immediately.
>>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>>> and
>>> > attachments are free from any virus, we advise that in keeping with
>>> good
>>> > computing practice the recipient should ensure they are actually virus
>>> free.
>>> >
>>>
>>>
>>> --
>>> Best
>>>
>>> ConradJam
>>>
>>> Confidentiality: This communication and any attachments are intended for
>>> the above-named persons only and may be confidential and/or legally
>>> privileged. Any opinions expressed in this communication are not
>>> necessarily those of NICE Actimize. If this communication has come to you
>>> in error you must take no action based on it, nor must you copy or show it
>>> to anyone; please delete/destroy and inform the sender by e-mail
>>> immediately.
>>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>>> Viruses: Although we have taken steps toward ensuring that this e-mail
>>> and attachments are free from any virus, we advise that in keeping with
>>> good computing practice the recipient should ensure they are actually virus
>>> free.
>>>
>>
Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)
Posted by Alexis Sarda-Espinosa <sa...@gmail.com>.
Hi Martijn,
just to be sure, if all state-related classes use a POJO serializer, Kryo
will never come into play, right? Given FLINK-16686 [1], I wonder how many
users actually have jobs with Kryo and RocksDB, but even if there aren't
many, that still leaves those who don't use RocksDB for
checkpoints/savepoints.
If Kryo were to stay in the Flink APIs in v1.X, is it impossible to let
users choose between v2/v5 jars by separating them like log4j2 jars?
[1] https://issues.apache.org/jira/browse/FLINK-16686
Regards,
Alexis.
Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
martijnvisser@apache.org>:
> Hi all,
>
> I also saw a thread on this topic from Clayton Wohl [1] on this topic,
> which I'm including in this discussion thread to avoid that it gets lost.
>
> From my perspective, there's two main ways to get to Java 17:
>
> 1. The Flink community agrees that we upgrade Kryo to a later version,
> which means breaking all checkpoint/savepoint compatibility and releasing a
> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala API support
> dropped. This is probably the quickest way, but would still mean that we
> expose Kryo in the Flink APIs, which is the main reason why we haven't been
> able to upgrade Kryo at all.
> 2. There's a contributor who makes a contribution that bumps Kryo, but
> either a) automagically reads in all old checkpoints/savepoints in using
> Kryo v2 and writes them to new snapshots using Kryo v5 (like is mentioned
> in the Kryo migration guide [2][3] or b) provides an offline tool that
> allows users that are interested in migrating their snapshots manually
> before starting from a newer version. That potentially could prevent the
> need to introduce a new Flink major version. In both scenarios, ideally the
> contributor would also help with avoiding the exposure of Kryo so that we
> will be in a better shape in the future.
>
> It would be good to get the opinion of the community for either of these
> two options, or potentially for another one that I haven't mentioned. If it
> appears that there's an overall agreement on the direction, I would propose
> that a FLIP gets created which describes the entire process.
>
> Looking forward to the thoughts of others, including the Users (therefore
> including the User ML).
>
> Best regards,
>
> Martijn
>
> [1] https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
> [2] https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>
> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
>
>> I agree, there are several options to mitigate the migration from v2 to
>> v5.
>> yet, Oracle roadmap is to end JDK 11 support in September this year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <ja...@gmail.com>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: Re: [Discussion] - Release major Flink version to support JDK 17
>> (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until I saw a
>> conversation in ISSUSE a few days ago and learned that the Kryo version
>> problem will affect the JDK17 compilation of snapshots [1] FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate from Kryo old
>> version checkpoint to the new Kryo new checkpoint at one time (Maybe this
>> tool start in flink 2.0 ?), does this tool currently have any plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with JDK11 and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version of Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to deprecate Java 8 instead
>> of requiring Java 11 https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <Ta...@niceactimize.com> 于2023年3月16日周四 00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion following this epic:
>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker now is Flink which
>> > currently remains on JDK 11 (LTS). Flink does not support JDK 17 yet,
>> with
>> > no timeline, the reason, based on the aforementioned ticket is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a major version (Flink
>> > 2.0.0) using the latest Kryo version for those who don't need to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will be handled only
>> > once - a mitigation can be achieved by a conversion tool or external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will be released this
>> > September. furthermore, Flink already supports JDK 12-15, which is very
>> > close to JDK 17 (LTS) - that was released in September 2021. JDK 11
>> will
>> > become a legacy soon, as more frameworks moving towards JDK 17 and are
>> less
>> > likely to support JDK 11 in the near future. (For example, Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments are intended for
>> > the above-named persons only and may be confidential and/or legally
>> > privileged. Any opinions expressed in this communication are not
>> > necessarily those of NICE Actimize. If this communication has come to
>> you
>> > in error you must take no action based on it, nor must you copy or show
>> it
>> > to anyone; please delete/destroy and inform the sender by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> > Viruses: Although we have taken steps toward ensuring that this e-mail
>> and
>> > attachments are free from any virus, we advise that in keeping with good
>> > computing practice the recipient should ensure they are actually virus
>> free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments are intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has come to you
>> in error you must take no action based on it, nor must you copy or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
>> Viruses: Although we have taken steps toward ensuring that this e-mail
>> and attachments are free from any virus, we advise that in keeping with
>> good computing practice the recipient should ensure they are actually virus
>> free.
>>
>