You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/02/21 18:44:34 UTC

[GitHub] [flink] StephanEwen opened a new pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

StephanEwen opened a new pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179
 
 
   ## NOTE: This removes savepoint compatibility with Flink 1.2 !
   
   For background, please consult [FLINK-16192](https://issues.apache.org/jira/browse/FLINK-16192)
   
   ## What is the purpose of the change
   
   **This is part of the preparation for implementing Fault Tolerance for Operator Coordinators for the new Source Interface (FLIP-27).**
   
   This PR contains various cleanups / simplifications / refactorings around the checkpoint metadata serialization / deserialization code, to make it easier to implement the handling of Operator Coordinator State.
   
   ...and to finally drop some really old and unused code. I felt a bit like an archaeologist digging through this ;-)
   
   
   ## Brief change log
   
   The main changes are:
   
     - 5436317bc99830bdc21b5799fcabf8f833120932 : *[FLINK-16178][refactor] Make the versioned Checkpoint Metadata Serializers only responsible for deserialization.*
   
       The metadata is always serialized with the latest serializer. Deserialization uses a version-specific serializer. That is the same functionality as before, but cleanly represented in the code. The code previously pretended to be version-dynamic on serialization side as well, but still could only handle the latest version on serialization. Earlier versions failed with a `ClassCastException` or `UnsupportedMethodException`.
   
       Weird constructs were necessary, like a `SavepointV1Serializer implements SavepointSerializer<SavepointV2>` (mismatch between handled version and generic type).
   
     - 247bd519b8609610c80704bb4a7c0e5a8375494a : *[FLINK-16192][checkpointing] Remove remaining bits of "legacy state" and remove Savepoint 1.2 compatibility*
   
       "Legacy State" Refers to the state originally created by the old `Checkpointed` interface, before state was re-scalable. This was later replaced by `CheckpointedFunction` (and `ListCheckpointed` as the shortcut).
   
       This state was no longer supported since Flink 1.4. However, all 1.2 savepoints in migration tests (and some 1.3 savepoints for tests) used that "legacy state", that is why the code was retained and support for "legacy state" was hackily activated (via some static flags) for migration tests.
   
       Removing that code would actually not reduce any savepoint compatibility, because compatibility for 1.2 and 1.3 was anyways only given for jobs that did not use that "legacy state". However, removing this code makes all 1.2 savepoints from tests unusable.
   
       A [mailing list discussion](https://lists.apache.org/x/thread.html/rd36579fe754f7557da4ae1dda47820c84b21009c35e6e317671e83b0@%3Cdev.flink.apache.org%3E) suggested to drop that support, tather than elaborate re-building of these the test data for Flink 1.2 with different state type.
   
   ## Verifying this change
   
   This change is already covered by existing tests, such as the savepoint tests and savepoint migration tests.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): **no**
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
     - The serializers: **no**
     - The runtime per-record code paths (performance sensitive): **no**
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **yes**
     - The S3 file system connector: **no**
   
   ## Documentation
   
     - Does this pull request introduce a new feature? **no**
     - If yes, how is the feature documented? **not applicable**
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-589793675
 
 
   <!--
   Meta data
   Hash:247bd519b8609610c80704bb4a7c0e5a8375494a Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/150081947 TriggerType:PUSH TriggerID:247bd519b8609610c80704bb4a7c0e5a8375494a
   -->
   ## CI report:
   
   * 247bd519b8609610c80704bb4a7c0e5a8375494a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/150081947) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] buptljy commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
buptljy commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r382885940
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   Should we bind #SavepointSerializer to #SavepointV2 ? 
   In this way, we have to create a new interface if we want to add #SavepointV3 in the future. Is that expected?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-589793675
 
 
   <!--
   Meta data
   Hash:247bd519b8609610c80704bb4a7c0e5a8375494a Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:247bd519b8609610c80704bb4a7c0e5a8375494a
   -->
   ## CI report:
   
   * 247bd519b8609610c80704bb4a7c0e5a8375494a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] Jiayi-Liao commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
Jiayi-Liao commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383065779
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   You're right. I was misguided by the previous interface. Would it be better if we change the function name to "deserializeToLatestVersion" ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen closed pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen closed pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-589783809
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 247bd519b8609610c80704bb4a7c0e5a8375494a (Fri Feb 21 18:47:30 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-589793675
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "247bd519b8609610c80704bb4a7c0e5a8375494a",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/150081947",
       "triggerID" : "247bd519b8609610c80704bb4a7c0e5a8375494a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "247bd519b8609610c80704bb4a7c0e5a8375494a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5440",
       "triggerID" : "247bd519b8609610c80704bb4a7c0e5a8375494a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 247bd519b8609610c80704bb4a7c0e5a8375494a Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/150081947) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5440) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-589793675
 
 
   <!--
   Meta data
   Hash:247bd519b8609610c80704bb4a7c0e5a8375494a Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/150081947 TriggerType:PUSH TriggerID:247bd519b8609610c80704bb4a7c0e5a8375494a
   Hash:247bd519b8609610c80704bb4a7c0e5a8375494a Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5440 TriggerType:PUSH TriggerID:247bd519b8609610c80704bb4a7c0e5a8375494a
   -->
   ## CI report:
   
   * 247bd519b8609610c80704bb4a7c0e5a8375494a Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/150081947) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5440) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-590814924
 
 
   @klion26 Thank you for taking a look! Your suggestions are implemented in this follow-up PR: #11202
   
   There is only one "Savepoint" class any more (renamed to "CheckpointMetadata") and it is not versioned. Only the serializers are versions.
   
   I would merge the two PRs together, because the second PR finishes the work of the first. I was only trying to spit it into different PRs so they would not grow too big.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] buptljy commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
buptljy commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r382885940
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   Should we bind #SavepointSerializer to #SavepointV2 ? 
   In this way, we have to create a new interface if we want to add #SavepointV3 in the future. Is this expected?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] klion26 commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
klion26 commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383730724
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializers.java
 ##########
 @@ -35,7 +35,7 @@
 	private static final Map<Integer, SavepointSerializer<?>> SERIALIZERS = new HashMap<>(2);
 
 	static {
-		SERIALIZERS.put(SavepointV1.VERSION, SavepointV1Serializer.INSTANCE);
+		SERIALIZERS.put(SavepointV1Serializer.VERSION, SavepointV1Serializer.INSTANCE);
 		SERIALIZERS.put(SavepointV2.VERSION, SavepointV2Serializer.INSTANCE);
 
 Review comment:
   Seems we just keep the recent savepoint, I'm thinking can we replacing all `SavepointV2` with `Savepoint` directly, after the replacement, we just have one `Savepoint` class, no `SavepointVX` class, and update the `VERSION` property in `Savepoint` when upgrading.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-590751688
 
 
   The next step of cleanups is in #11202 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-589793675
 
 
   <!--
   Meta data
   Hash:247bd519b8609610c80704bb4a7c0e5a8375494a Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/150081947 TriggerType:PUSH TriggerID:247bd519b8609610c80704bb4a7c0e5a8375494a
   Hash:247bd519b8609610c80704bb4a7c0e5a8375494a Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5440 TriggerType:PUSH TriggerID:247bd519b8609610c80704bb4a7c0e5a8375494a
   -->
   ## CI report:
   
   * 247bd519b8609610c80704bb4a7c0e5a8375494a Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/150081947) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5440) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383416900
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   The related PR is #11202 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383013553
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   This has always been the case in the code. Previously, the interface "pretended" to be cross version generic, but in the end all deserializers had to be updated every time a new version came. This now makes this explicit.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-590751921
 
 
   @carp84 I see you assigned yourself as a reviewer. Do you plan to give this a pass or is this okay from your side?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] Jiayi-Liao commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
Jiayi-Liao commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383065779
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   You're right. I was misguided by the previous interface. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] klion26 commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
klion26 commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383731462
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.java
 ##########
 @@ -103,7 +102,7 @@ public static Savepoint loadCheckpointMetadata(DataInputStream in, ClassLoader c
 
 
 Review comment:
   maybe we can replace `Savepoint` with `SavepointV2` on line 97.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-591117477
 
 
   Merged in b8277f92cfd1fa80325194bd705d1a13293b30e3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen edited a comment on issue #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#issuecomment-590814924
 
 
   @klion26 Thank you for taking a look! Your suggestions are implemented in this follow-up PR: #11202
   
   There is only one "Savepoint" class any more (renamed to "CheckpointMetadata") and it is not versioned. Only the serializers are versioned.
   
   I would merge the two PRs together, because the second PR finishes the work of the first. I was only trying to spit it into different PRs so they would not grow too big.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383013478
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   In this design, this is expected. If we introduce a new format version of the metadata, all deserializers need to deserialize to that new version.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #11179: [FLINK-16178][FLINK-16192][checkpointing] Clean up checkpoint metadata code and remove remaining bits of "legacy state"
URL: https://github.com/apache/flink/pull/11179#discussion_r383383470
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointSerializer.java
 ##########
 @@ -52,6 +37,5 @@
 	 * @return The deserialized savepoint
 	 * @throws IOException Serialization failures are forwarded
 	 */
-	T deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
-
+	SavepointV2 deserialize(DataInputStream dis, ClassLoader userCodeClassLoader) throws IOException;
 
 Review comment:
   This change here contains something similar to your suggestion: https://issues.apache.org/jira/browse/FLINK-16259

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services