You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "kurtostfeld (via GitHub)" <gi...@apache.org> on 2023/05/25 16:59:52 UTC

[GitHub] [flink] kurtostfeld opened a new pull request, #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

kurtostfeld opened a new pull request, #22660:
URL: https://github.com/apache/flink/pull/22660

   …ith backward compatibility for existing savepoints and checkpoints.
   
   ## What is the purpose of the change
   
   To upgrade the primary Kryo library used by Flink from v2.x to v5.x, while providing backwards compatibility with existing savepoints and checkponts. This PR adds a new Kryo v5 dependency that is namespaced so that it can coexist with the legacy dependencies that would be kept for compatibility purposes. The existing chill-java library version 0.7.6 would also be deprecated and kept for compatibility purposes only. Some future version of Flink could eventually drop Kryo v2 and chill-java when backwards compatibility with Kryo v2 based state is no longer needed.
   
   Why upgrade Kryo:
   
   - Kryo v5.x is more compatible with newer versions of Java. I wrote a simple Flink app with Kryo serialized state: When running under Java 17 or Java 21 pre-release builds, the Kryo v2 library fails at runtime, while a prototype build of Flink with this PR can load state serialized with Kryo v5.x running under both Java 17 and Java 21 pre-release builds successfully without failures.
   - Kryo v5.x supports Java records, while Kryo v2.x does not.
   - Kryo v2.2.x was released over ten years ago, well before the release of Java 8. Kryo is a maintained project, there have been lots of improvements over the past ten years. Kryo v5 has faster runtime performance, and more memory efficient serialization protocols. Additionally Kryo v5 has fixed lots of bugs. While the Flink project has worked around all Kryo v2 bugs, it would be an improvement to remove the workarounds. Next, Kryo v2 depends on the chill-java library for functionality, where almost all of that functionality is built into Kryo v5.x so that chill-java dependency can be phased out.
   
   ## Brief change log
   
   This is a large PR with a lot of surface area and risk. I tried to keep the scope of these changes as narrow and simple as possible. I copied all the Kryo v2 code to Kryo v5 equivalents and made necessary adjustments to get everything working. Some Flink serialization code references Flink Java classes by full package name, so I didn't modify the package names or class names of any existing serialization classes.
   
   ## Verifying this change
   
   There are lots of existing unit tests that cover backward compatibility with existing state and also the serialization framework. This obviously passes all those tests. Additionally, I wrote a Flink application to do a more thorough test of the Kryo upgrade that was difficult to convert into unit test form.
   
   https://github.com/kurtostfeld/flink-kryo-upgrade-demo
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): yes. This adds a new Kryo v5 dependency and keeps legacy dependencies for backwards compatibility purposes.
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes. Kryo v2 APIs are deprecated. Parallel Kryo v5 APIs are created with PublicEvolving
     - The serializers: yes. This absolutely affects the serializers.
     - The runtime per-record code paths (performance sensitive): yes. This should be faster but I haven't done any benchmark testing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w… [flink]

Posted by "holleyross80-ai (via GitHub)" <gi...@apache.org>.

holleyross80-ai commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-3977199688

   Everything will check in coming access I'll be URL and and recorded for data some as any outgoing double checked make sure it's going to Wikipedia wherever it says it's going to use the CC licence by loading function and voiceover gestures bleachers are saying for two Jason code co-pilot and more I'll get fixing them now thank you everyone for your patience also would like Sandra in the background backup portion of her any sign we get breached anything gets access without our permission of the operating system knowing to activate sandtrap 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563767323

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563804659

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w… [flink]

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld closed pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…
URL: https://github.com/apache/flink/pull/22660


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.

MartijnVisser commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1567897701

   > Could I get permissions to do this? Thank you :)
   
   You should have permissions by now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "zentol (via GitHub)" <gi...@apache.org>.

zentol commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1564201206

   I appreciate the work, but really this change has to go through a proper design process.
   
   There are open questions as to whether we really want to expose Kryo in the same way we did before (because the presence of Kryo serializers in the execution config is a problem in general).
   
   Please create a [FLIP](https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals) and publish it on the dev mailing list.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.

flinkbot commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563235211

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d24bb0737d2a7b4b61f51843241ab78a47302d0a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d24bb0737d2a7b4b61f51843241ab78a47302d0a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d24bb0737d2a7b4b61f51843241ab78a47302d0a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563797525

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563776356

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] nicknezis commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "nicknezis (via GitHub)" <gi...@apache.org>.

nicknezis commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1639043540

   FYI, there is a pull request in the works to upgrade Twitter Chill to use Kryo 5.5.0. I don't think this PR is blocked by the Chill pull request, but might be relevant, so I'm sharing the link. https://github.com/twitter/chill/pull/747


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1564710396

   @zentol ok, I created a Confluence login with username "kurto", however I don't seem to have permissions to create a new FLIP. I don't see a "Create" button as the docs say:
   
   > To create your own FLIP, click on "Create" on the header and choose "FLIP-Template" other than "Blank page".
   
   Thank you :)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563366271

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] kurtostfeld commented on pull request #22660: [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w…

Posted by "kurtostfeld (via GitHub)" <gi...@apache.org>.

kurtostfeld commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-1563370518

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-3154][runtime] Upgrade from Kryo v2 + Chill 0.7.6 to Kryo v5 w… [flink]

Posted by "nicknezis (via GitHub)" <gi...@apache.org>.

nicknezis commented on PR #22660:
URL: https://github.com/apache/flink/pull/22660#issuecomment-2310424567

   I added my comment on the [Jira ticket](https://issues.apache.org/jira/browse/FLINK-3154), but also posting it here for better visibility.
   
   I wanted to revisit the status of this ticket. There was some discussion in the [mailing list](https://lists.apache.org/thread/6lv5b0q3l1fjhov3txrd3skk3ldlgo3g) about merging this into Flink 2.0. If the master branch now is targeting 2.0, maybe we can finally move forward with this [FLIP](https://cwiki.apache.org/confluence/display/FLINK/FLIP-317:+Upgrade+Kryo+from+2.24.0+to+5.5.0)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org