[jira] [Commented] (FLINK-11591) Restoring LockableTypeSerializer's snapshot from 1.6 and below requires pre-processing before snapshot is valid to use

["Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768029#comment-16768029 ] 

Tzu-Li (Gordon) Tai commented on FLINK-11591:
---------------------------------------------

Merged for 1.8.0 - f9532cba36c04fbde7e6956a0813ba260ad96953


> Restoring LockableTypeSerializer's snapshot from 1.6 and below requires pre-processing before snapshot is valid to use
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-11591
>                 URL: https://issues.apache.org/jira/browse/FLINK-11591
>             Project: Flink
>          Issue Type: Bug
>          Components: CEP, Type Serialization System
>    Affects Versions: 1.7.0, 1.7.1
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Igal Shilman
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.8.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> In 1.6 and below, the {{LockableTypeSerializer}} incorrectly returns directly the element serializer's snapshot instead of wrapping it within an independent snapshot class:
> https://github.com/apache/flink/blob/release-1.6/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/Lockable.java#L188
> This results in the fact that the the written state information for this would be {{(LockableTypeSerializer, SomeArbitrarySnapshot)}}.
> The problem occurs when restoring this in Flink 1.7+, since compatibility checks are now performed by providing the new serializer to the snapshot, what would happen is:
> {{SomeArbitrarySnapshot.resolveSchemaCompatibility(newLockableTypeSerializer)}}, which would not work since the arbitrary snapshot does not recognize the {{LockableTypeSerializer}}.
> To fix this, we essentially need to preprocess that arbitrary snapshot when restoring from <= 1.6 version snapshots.
> A proposed fix would be to have the following interface:
> {code}
> public interface RequiresLegacySerializerSnapshotPreprocessing<T> {
>     TypeSerializerSnapshot<T> preprocessLegacySerializerSnapshot(TypeSerializerSnapshot<T> legacySnapshot)
> }
> {code}
> The {{LockableTypeSerializer}} would then implement this interface, and in the {{preprocessLegacySerializerSnapshot}} method, properly wrap that arbitrary element serializer snapshot into a {{LockableTypeSerializerSnapshot}}.
> In general, this interface is useful to preprocess any problematic snapshot that was returned pre 1.7.
> The point-in-time to check if a written serializer in <= 1.6 savepoints implements this interface and preprocesses the read snapshot would be:
> https://github.com/apache/flink/blob/a567a1ef628eadad21e11864ec328481cd6d7898/flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializerSerializationUtil.java#L218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)