You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2017/04/17 06:40:42 UTC
[jira] [Comment Edited] (FLINK-5697) Add per-shard watermarks for
FlinkKinesisConsumer
[ https://issues.apache.org/jira/browse/FLINK-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970738#comment-15970738 ]
Tzu-Li (Gordon) Tai edited comment on FLINK-5697 at 4/17/17 6:40 AM:
---------------------------------------------------------------------
I'd say it still makes sense. If I understand how Kinesis resharding works correctly, merging or splitting shards actually just results in new empty shards with new key ranges, while the previous shards that were merged or split are simple closed (i.e., records that were collected by Kinesis before the resharding are consumed from the old closed shards). Therefore, we can still assume simple event time patterns per-shard, as there is no merging or splitting of the Kinesis records taking place.
was (Author: tzulitai):
I'd say it still makes sense. If I understand how Kinesis resharding works correctly, merging or splitting shards always result in new empty shards with new key ranges, and the previous shards that were merged or split are simple closed (i.e., records that were collected by Kinesis before the resharding are consumed from the old closed shards). Therefore, we can still assume simple event time patterns per-shard, as there is no merging or splitting of the Kinesis records taking place.
> Add per-shard watermarks for FlinkKinesisConsumer
> -------------------------------------------------
>
> Key: FLINK-5697
> URL: https://issues.apache.org/jira/browse/FLINK-5697
> Project: Flink
> Issue Type: New Feature
> Components: Kinesis Connector, Streaming Connectors
> Reporter: Tzu-Li (Gordon) Tai
>
> It would be nice to let the Kinesis consumer be on-par in functionality with the Kafka consumer, since they share very similar abstractions. Per-partition / shard watermarks is something we can add also to the Kinesis consumer.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)