You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2022/10/31 09:57:00 UTC
[jira] [Updated] (FLINK-29807) Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
[ https://issues.apache.org/jira/browse/FLINK-29807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nowojski updated FLINK-29807:
-----------------------------------
Description:
The motivation behind this move is two fold. One reason is that it complicates our code base unnecessarily and creates confusion on how to actually implement custom serializers. The immediate reason is that I wanted to clean up Flink's configuration stack a bit and refactor the ExecutionConfig class FLINK-29379. This refactor would keep the API compatibility of the ExecutionConfig, but it would break savepoint compatibility with snapshots written with some of the old serializers, which had ExecutionConfig as a field and were serialized in the snapshot. This issue has been resolved by the introduction of TypeSerializerSnapshot in Flink 1.7 FLINK-9377, where serializers are no longer part of the snapshot.
TypeSerializerConfigSnapshot has been deprecated and no longer used by built-in serializers since Flink 1.8 FLINK-9376 and FLINK-11323. Users were encouraged to migrate to TypeSerializerSnapshot since then with their own custom serializers. That has been plenty of time for the migration.
This proposal would have the following impact for the users:
1. we would drop support for recovery from savepoints taken with Flink < 1.7.0 for all built in types serializers
2. we would drop support for recovery from savepoints taken with Flink < 1.8.0 for built in kryo serializers
3. we would drop support for recovery from savepoints taken with Flink < 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot
1. and 2. would have a simple migration path. Users migrating from those old savepoints would have to first start his job using a Flink version from the [1.8, 1.16] range, and take a new savepoint that would be compatible with Flink 1.17.
3. This is a bit more problematic, because users would have to first migrate their own custom serializers to use TypeSerializerSnapshot (using a Flink version from the [1.8, 1.16]), take a savepoint, and only then migrate to Flink 1.17. However users had already 4 years to migrate, which in my opinion has been plenty of time to do so.
*As discussed and vote is currently in progress:* https://lists.apache.org/thread/x5d0p08pf2wx47njogsgqct0k5rpfrl4
was:
The motivation behind this move is two fold. One reason is that it complicates our code base unnecessarily and creates confusion on how to actually implement custom serializers. The immediate reason is that I wanted to clean up Flink's configuration stack a bit and refactor the ExecutionConfig class [2]. This refactor would keep the API compatibility of the ExecutionConfig, but it would break savepoint compatibility with snapshots written with some of the old serializers, which had ExecutionConfig as a field and were serialized in the snapshot. This issue has been resolved by the introduction of TypeSerializerSnapshot in Flink 1.7 [3], where serializers are no longer part of the snapshot.
TypeSerializerConfigSnapshot has been deprecated and no longer used by built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged to migrate to TypeSerializerSnapshot since then with their own custom serializers. That has been plenty of time for the migration.
This proposal would have the following impact for the users:
1. we would drop support for recovery from savepoints taken with Flink < 1.7.0 for all built in types serializers
2. we would drop support for recovery from savepoints taken with Flink < 1.8.0 for built in kryo serializers
3. we would drop support for recovery from savepoints taken with Flink < 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot
1. and 2. would have a simple migration path. Users migrating from those old savepoints would have to first start his job using a Flink version from the [1.8, 1.16] range, and take a new savepoint that would be compatible with Flink 1.17.
3. This is a bit more problematic, because users would have to first migrate their own custom serializers to use TypeSerializerSnapshot (using a Flink version from the [1.8, 1.16]), take a savepoint, and only then migrate to Flink 1.17. However users had already 4 years to migrate, which in my opinion has been plenty of time to do so.
*As discussed and vote is currently in progress*: https://lists.apache.org/thread/x5d0p08pf2wx47njogsgqct0k5rpfrl4
> Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
> -----------------------------------------------------------------------------------
>
> Key: FLINK-29807
> URL: https://issues.apache.org/jira/browse/FLINK-29807
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.17
> Reporter: Piotr Nowojski
> Assignee: Piotr Nowojski
> Priority: Major
> Fix For: 1.17
>
>
> The motivation behind this move is two fold. One reason is that it complicates our code base unnecessarily and creates confusion on how to actually implement custom serializers. The immediate reason is that I wanted to clean up Flink's configuration stack a bit and refactor the ExecutionConfig class FLINK-29379. This refactor would keep the API compatibility of the ExecutionConfig, but it would break savepoint compatibility with snapshots written with some of the old serializers, which had ExecutionConfig as a field and were serialized in the snapshot. This issue has been resolved by the introduction of TypeSerializerSnapshot in Flink 1.7 FLINK-9377, where serializers are no longer part of the snapshot.
> TypeSerializerConfigSnapshot has been deprecated and no longer used by built-in serializers since Flink 1.8 FLINK-9376 and FLINK-11323. Users were encouraged to migrate to TypeSerializerSnapshot since then with their own custom serializers. That has been plenty of time for the migration.
> This proposal would have the following impact for the users:
> 1. we would drop support for recovery from savepoints taken with Flink < 1.7.0 for all built in types serializers
> 2. we would drop support for recovery from savepoints taken with Flink < 1.8.0 for built in kryo serializers
> 3. we would drop support for recovery from savepoints taken with Flink < 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot
> 1. and 2. would have a simple migration path. Users migrating from those old savepoints would have to first start his job using a Flink version from the [1.8, 1.16] range, and take a new savepoint that would be compatible with Flink 1.17.
> 3. This is a bit more problematic, because users would have to first migrate their own custom serializers to use TypeSerializerSnapshot (using a Flink version from the [1.8, 1.16]), take a savepoint, and only then migrate to Flink 1.17. However users had already 4 years to migrate, which in my opinion has been plenty of time to do so.
> *As discussed and vote is currently in progress:* https://lists.apache.org/thread/x5d0p08pf2wx47njogsgqct0k5rpfrl4
--
This message was sent by Atlassian Jira
(v8.20.10#820010)