[jira] [Resolved] (FLINK-6425) Integrate serializer reconfiguration into state restore flow to activate serializer upgrades

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tzu-Li (Gordon) Tai resolved FLINK-6425.
----------------------------------------
    Resolution: Fixed

Merged for 1.3.0 with https://git-wip-us.apache.org/repos/asf/flink/commit/63c04a5

> Integrate serializer reconfiguration into state restore flow to activate serializer upgrades
> --------------------------------------------------------------------------------------------
>
>                 Key: FLINK-6425
>                 URL: https://issues.apache.org/jira/browse/FLINK-6425
>             Project: Flink
>          Issue Type: Sub-task
>          Components: State Backends, Checkpointing
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>
> With FLINK-6191, {{TypeSerializer}} will be reconfigurable.
> From the state backends' point of view, serializer reconfiguration doubles as a mechanism to determine how serializer upgrades should be handled.
> The general idea is that state checkpoints should contain the following as the state's metainfo:
> - the previous serializer
> - snapshot of the previous serializer's configuration
> The upgrade flow is as follows:
> 1. On restore, try to deserialize the previous old serializer. Deserialization may fail if a) the serializer no longer exists in classpath, or b) the serializer class is not longer valid (i.e., implementation changed and resulted in different serialVersionUID). In this case, use a dummy serializer as a placeholder. This dummy serializer is currently the {{ClassNotFoundProxySerializer}} in the code.
> 2. Deserialize the configuration snapshot of the previous old serializer. The configuration snapshot must be successfully deserialized, otherwise the state restore fails.
> 3. When we get the new registered serializer for the state (could be a completely new serializer, the same serializer with different implementations, or the exact same serializer untouched; either way they are seen as a new serializer), we use the configuration snapshot of the old serializer to reconfigure the new serializer.
> This completes the upgrade of the old serializer.  However, depending on the result of the upgrade, state conversion needs to take place (for now, if state conversion is required, we just fail the job as this functionality isn't available yet). The results could be:
> - Compatible: restore success + serializer upgraded.
> - Compatible, but serialization schema changed: serializer upgraded but requires state conversion, without the requirement that the old serializer needs to be present.
> - Incompatible: serializer upgraded requires state conversion, but requires the old serializer to be present (i.e., can not be the dummy {{ClassNotFoundProxySerializer}}).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)