[jira] [Updated] (FLINK-37551) Improve the state compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend

["Ruan Hang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-37551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruan Hang updated FLINK-37551:
------------------------------
    Fix Version/s: 2.3.0
                       (was: 2.2.0)

> Improve the state compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend
> --------------------------------------------------------------------------------------
>
>                 Key: FLINK-37551
>                 URL: https://issues.apache.org/jira/browse/FLINK-37551
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 2.0.0
>            Reporter: Yue Ma
>            Assignee: Yue Ma
>            Priority: Major
>             Fix For: 2.3.0
>
>
> Currently, when switching between ForStKeyeStateBackend and ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a lot of upgrade costs for users. The main purpose of this ticket is to improve the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.
> In order to support the sync/async mode compatibility between ForStKeyedStateBackend, there may be many details involved. For the convenience of future tracking, the required changes will be summarized here. (The current list may still be incomplete, but it will be continuously updated until full compatibility)
>  * {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and ForStSyncKeyeStateBackend{*}.  Currently, if MapState or ListState is used, the StateMeta information saved by ForStKeyeStateBackend and ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause incompatible states. For example, when using MapState, ForStKeyeStateBackend will save the ValueSerializer information in StateMeta, but ForStSyncKeyeStateBackend will save the MapSerializer information. So we need to unify this place and perform a transformation when ForStKeyeStateBackend creates stateMeta
>  * {*}Make ForStKeyedStateBackend consider namespaceSerializer compatibility when restoring state{*}. ForStKeyedStateBackend currently does not consider namespaceSerializer compatibility when restoring state. This may cause the namespaceSerializer to change when it is not known whether it should be compatible.
>  * {*}Make StateMeta in ForState restore from oldStateInfo from  Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently, when creating using ForStKeyedStateBackend create a new State , AbstractKeyedState will directly use the serializer in StateDescriptor regardless of whether it is restored from a Checkpoint.However, this may cause data to be unable to be restored when the last serializer changes (for example, the class registration order in KryoSerializer has changed, for specific cases, please refer toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder). Therefore, when creating a State from a Checkpoint, oldStateInfo should be used as the new State's Serializer.
>  * *Add unit tests for state compatibility between ForStKeyeStateBackend and ForStSyncKeyeStateBackend*
>  * *Make serialization consistent between ForStMapState and ForStSyncMapState* (https://issues.apache.org/jira/browse/FLINK-37419)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)