[jira] [Commented] (FLINK-11947) Support MapState key / value schema evolution for RocksDB

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825795#comment-16825795 ] 

Tzu-Li (Gordon) Tai commented on FLINK-11947:
---------------------------------------------

For a first step, I agree that we should at least first make the schema evolution limitation exclusive to the user key of MapState.
This should allow us to use the current way migration is performed in the RocksDB state backend to have state schema evolution for user values.

> Support MapState key / value schema evolution for RocksDB
> ---------------------------------------------------------
>
>                 Key: FLINK-11947
>                 URL: https://issues.apache.org/jira/browse/FLINK-11947
>             Project: Flink
>          Issue Type: Improvement
>          Components: API / Type Serialization System, Runtime / State Backends
>            Reporter: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>
> Currently, we do not attempt to perform state schema evolution if the key or value's schema of a user {{MapState}} has changed when using {{RocksDB}}:
> https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542
> This was disallowed in the initial support for state schema evolution because the way we did state evolution in the RocksDB state backend was simply overwriting values.
> For {{MapState}} key evolution, only overwriting RocksDB values does not work, since RocksDB entries for {{MapState}} uses a composite key containing the map state key. This means that when evolving {{MapState}} in this case with an evolved key schema, we will have new entries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)