You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zoltar9264 (via GitHub)" <gi...@apache.org> on 2023/08/05 10:16:31 UTC

[GitHub] [flink] zoltar9264 opened a new pull request, #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

zoltar9264 opened a new pull request, #23146:
URL: https://github.com/apache/flink/pull/23146

   … as SharedStateRegistryKey ChangelogStateHandleStreamImpl
   
   ## What is the purpose of the change
   
   As title, details in [FLINK-32761](https://issues.apache.org/jira/browse/FLINK-32761).
   
   ## Brief change log
   
     - move *IncrementalRemoteKeyedStateHandle#createSharedStateRegistryKey()* to *SharedStateRegistry*
     - replace *ChangelogStateHandleStreamImpl#getKey()* with *SharedStateRegistry#createSharedStateRegistryKey()*
   
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zoltar9264 commented on pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "zoltar9264 (via GitHub)" <gi...@apache.org>.
zoltar9264 commented on PR #23146:
URL: https://github.com/apache/flink/pull/23146#issuecomment-1677831666

   Thanks for reply @rkhachatryan , and can you help merge this pr ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] rkhachatryan merged pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan merged PR #23146:
URL: https://github.com/apache/flink/pull/23146


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] rkhachatryan commented on a diff in pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #23146:
URL: https://github.com/apache/flink/pull/23146#discussion_r1285261372


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/SharedStateRegistry.java:
##########
@@ -106,4 +108,12 @@ StreamStateHandle registerReference(
     void registerAllAfterRestored(CompletedCheckpoint checkpoint, RestoreMode mode);
 
     void checkpointCompleted(long checkpointId);
+
+    /** Create a unique key based on physical id. */
+    static SharedStateRegistryKey createSharedStateRegistryKey(StreamStateHandle handle) {
+        String keyString = handle.getStreamStateHandleID().getKeyString();
+        // key strings tend to be longer, so we use the MD5 of the key string to save memory
+        return new SharedStateRegistryKey(
+                UUID.nameUUIDFromBytes(keyString.getBytes(StandardCharsets.UTF_8)).toString());
+    }

Review Comment:
   Static factory methods are usually placed in the type which instance they create,
   so how about moving this method to `SharedStateRegistryKey`?
   And naming it something like `SharedStateRegistryKey.forStreamStateHandle` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zoltar9264 commented on a diff in pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "zoltar9264 (via GitHub)" <gi...@apache.org>.
zoltar9264 commented on code in PR #23146:
URL: https://github.com/apache/flink/pull/23146#discussion_r1285492407


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/SharedStateRegistry.java:
##########
@@ -106,4 +108,12 @@ StreamStateHandle registerReference(
     void registerAllAfterRestored(CompletedCheckpoint checkpoint, RestoreMode mode);
 
     void checkpointCompleted(long checkpointId);
+
+    /** Create a unique key based on physical id. */
+    static SharedStateRegistryKey createSharedStateRegistryKey(StreamStateHandle handle) {
+        String keyString = handle.getStreamStateHandleID().getKeyString();
+        // key strings tend to be longer, so we use the MD5 of the key string to save memory
+        return new SharedStateRegistryKey(
+                UUID.nameUUIDFromBytes(keyString.getBytes(StandardCharsets.UTF_8)).toString());
+    }

Review Comment:
   Thanks for the guidance, fixed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23146:
URL: https://github.com/apache/flink/pull/23146#issuecomment-1666464287

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "c7827a9115d9f07467a42c6681277736cff1744d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c7827a9115d9f07467a42c6681277736cff1744d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c7827a9115d9f07467a42c6681277736cff1744d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] rkhachatryan commented on a diff in pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on code in PR #23146:
URL: https://github.com/apache/flink/pull/23146#discussion_r1285261372


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/SharedStateRegistry.java:
##########
@@ -106,4 +108,12 @@ StreamStateHandle registerReference(
     void registerAllAfterRestored(CompletedCheckpoint checkpoint, RestoreMode mode);
 
     void checkpointCompleted(long checkpointId);
+
+    /** Create a unique key based on physical id. */
+    static SharedStateRegistryKey createSharedStateRegistryKey(StreamStateHandle handle) {
+        String keyString = handle.getStreamStateHandleID().getKeyString();
+        // key strings tend to be longer, so we use the MD5 of the key string to save memory
+        return new SharedStateRegistryKey(
+                UUID.nameUUIDFromBytes(keyString.getBytes(StandardCharsets.UTF_8)).toString());
+    }

Review Comment:
   Static factory methods are usually placed in the type which instance they create,
   so how about moving this method to `SharedStateRegistryKey`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] rkhachatryan commented on pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "rkhachatryan (via GitHub)" <gi...@apache.org>.
rkhachatryan commented on PR #23146:
URL: https://github.com/apache/flink/pull/23146#issuecomment-1678555324

   Sure @zoltar9264, let's merge it after the 1.8 feature freeze (after the branch cut).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zoltar9264 commented on pull request #23146: [FLINK-32761][state/backends] Use UUID based on PhysicalStateHandleID…

Posted by "zoltar9264 (via GitHub)" <gi...@apache.org>.
zoltar9264 commented on PR #23146:
URL: https://github.com/apache/flink/pull/23146#issuecomment-1667381900

   > Just to double-check: the change does not break the compatibility with the current checkpoints, does it?
   
   Thanks @rkhachatryan for helping to review this PR, I think it is compatible with the current checkpoints : It only changes the key used to register _StreamStateHandle_ with _SharedStateRegistry_, and does not change any thing in checkpoint metadata. When using CLAIM/LEGACY mode to start a job from a retained checkpoint, the retained checkpoint needs to be registered to the _SharedStateRegistry_, but since the registry key will be recalculated at this time, there will be no compatibility issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org