[GitHub] tzulitai commented on issue #6711: [FLINK-9377] [core, state backends] Remove serializers from checkpoints

[GitBox <gi...@apache.org>]

tzulitai commented on issue #6711: [FLINK-9377] [core, state backends] Remove serializers from checkpoints
URL: https://github.com/apache/flink/pull/6711#issuecomment-427744344
 
 
   Thanks a lot for the detailed reviews and discussions, @dawidwys and @StephanEwen.
   
   @dawidwys and I went through some offline second-pass discussions on the suggestions, and we finally came up with this (implemented in the follow-up e2cbe99):
   
   1. To incorporate all the desired changes to the current `TypeSerializerConfigSnapshot` class (e.g. not explicitly setting the classloader, dump using the `VersionedIOReadableWritable` as base class, etc.), we came up with introducing `TypeSerializerSnapshot` which will eventually be the new snapshot interface once we completely remove `TypeSerializerConfigSnapshot`:
   
   ```
   @PublicEvolving
   public interface TypeSerializerSnapshot<T> {
       int getCurrentVersion();
       void write(DataOutputView out) throws IOException;
       void read(int readVersion, DataInputView in, ClassLoader userCodeClassLoader) throws IOException;
       TypeSerializer<T> restoreSerializer();
       <NS extends TypeSerializer<T>> TypeSerializerSchemaCompatibility<T, NS> resolveSchemaCompatibility(NS newSerializer);
   }
   ```
   
   If a serializer's snapshot is still the legacy `TypeSerializerConfigSnapshot`, then the prior serializer will always be written along with it so that the dummy `restoreSerializer()` factory method works with requiring the user for any changes. Once the user upgrades to directly implementing the `TypeSerializerSnapshot`, then the `restoreSerializer()` method will be strictly required to be implemented, and the prior serializer will no longer be written anymore.
   
   It is only possible to change a snapshot from `TypeSerializerConfigSnapshot` to `TypeSerializerSnapshot`, and not the other way around.
   
   This allows us to incrementally upgrade each serializer snapshot class currently in Flink.
   We can still release if only partially some of the serializers have been upgraded.
   
   2. The `BackwardsCompatibleConfigSnapshot` wrapper class is still required, so that for Flink version <= 1.2 (which only wrote the serializer instance and no snapshots), we can wrap the serializer instance inside a snapshot implementation.
   
   Overall, the new changes quite closely follows your suggestions, only with a different approach to how the new / old interfaces work together.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services