You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2017/05/22 05:27:04 UTC
[jira] [Updated] (FLINK-6653) Avoid directly serializing AWS's
Shard class in Kinesis consumer's checkpoints
[ https://issues.apache.org/jira/browse/FLINK-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tzu-Li (Gordon) Tai updated FLINK-6653:
---------------------------------------
Description:
Currently, the Kinesis consumer's checkpoints directly serialize AWS's {{Shard}} instances in checkpoints. This makes bumping AWS library versions hard, since any change to the {{Shard}} class by AWS will break checkpoint compatibility.
We should either have custom serialization for {{KinesisStreamShard}}, or disintegrate the information in {{Shard}}. Ideally, it would be best to make {{KinesisStreamShard}} and {{SequenceNumber}} to be non-serializable, hence avoiding Java serialization in the checkpoints.
was:
Currently, the Kinesis consumer's checkpoints directly serialize AWS's `Shard` instances in checkpoints. This makes bumping AWS library versions hard, since any change to the `Shard` class by AWS will break checkpoint compatibility.
We should either have custom serialization for `KinesisStreamShard`, or disintegrate the information in `Shard`. Ideally, it would be best to make `KinesisStreamShard` and `SequenceNumber` to be non-serializable, hence avoiding Java serialization in the checkpoints.
> Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints
> ------------------------------------------------------------------------------
>
> Key: FLINK-6653
> URL: https://issues.apache.org/jira/browse/FLINK-6653
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
> Reporter: Tzu-Li (Gordon) Tai
>
> Currently, the Kinesis consumer's checkpoints directly serialize AWS's {{Shard}} instances in checkpoints. This makes bumping AWS library versions hard, since any change to the {{Shard}} class by AWS will break checkpoint compatibility.
> We should either have custom serialization for {{KinesisStreamShard}}, or disintegrate the information in {{Shard}}. Ideally, it would be best to make {{KinesisStreamShard}} and {{SequenceNumber}} to be non-serializable, hence avoiding Java serialization in the checkpoints.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)