[jira] [Commented] (FLINK-23143) Support state migration

["Hangxiang Yu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533756#comment-17533756 ] 

Hangxiang Yu commented on FLINK-23143:
--------------------------------------

[~roman] 
Sure, I just pushed my draft as you could see.

Another problem is the key serializer:
 # If the materialization part is empty, the key serializer will not be updated/check-compatibility. But the key serializer will be used to deserialize state in non-materialization part which will cause exception if the key serializer have changed.
 # For the non-materialization part, we may need to store its key serializer when snapshot and read the key serializer before build the delegated keyed state backend and changelog keyed state backend. But in the solution, we may need to change the process of {{ChangelogBackendRestoreOperation#restore}} .

Do you have any other better ideas about it ? Thanks a lot!

> Support state migration
> -----------------------
>
>                 Key: FLINK-23143
>                 URL: https://issues.apache.org/jira/browse/FLINK-23143
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / State Backends
>            Reporter: Roman Khachatryan
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 1.16.0
>
>
> ChangelogKeyedStateBackend.getOrCreateKeyedState is currently used during recovery; on 1st user access, it doesn't update metadata nor migrate state (as opposed to other backends).
>  
> The proposed solution is to
>  # wrap serializers (and maybe other objects) in getOrCreateKeyedState
>  # store wrapping objects in a new map keyed by state name
>  # pass wrapped objects to delegatedBackend.createInternalState
>  # on 1st user access, lookup wrapper and upgrade its wrapped serializer
> This should be done for both KV/PQ states.
>  
> See also [https://github.com/apache/flink/pull/15420#discussion_r656934791]
>  
> cc: [~yunta]



--
This message was sent by Atlassian Jira
(v8.20.7#820007)