[jira] [Commented] (FLINK-5697) Add per-shard watermarks for FlinkKinesisConsumer

["Thomas Weise (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669566#comment-16669566 ] 

Thomas Weise commented on FLINK-5697:
-------------------------------------

The latest related discussion: [https://lists.apache.org/thread.html/2341ab54f5c9a4f43f123605d6b17e374c19587bfd72c22748e08652@%3Cdev.flink.apache.org%3E]

I'm planning to first bring over the source watermarking that we implemented a while ago at Lyft to the Flink Kinesis consumer. Following that, we are planning to contribute the emit queue pattern as described in Jamie's document and the state sharing mechanism. Those pieces will allow to back pressure the shard consumer threads based on the shared watermark information. 

 

> Add per-shard watermarks for FlinkKinesisConsumer
> -------------------------------------------------
>
>                 Key: FLINK-5697
>                 URL: https://issues.apache.org/jira/browse/FLINK-5697
>             Project: Flink
>          Issue Type: New Feature
>          Components: Kinesis Connector, Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Thomas Weise
>            Priority: Major
>
> It would be nice to let the Kinesis consumer be on-par in functionality with the Kafka consumer, since they share very similar abstractions. Per-partition / shard watermarks is something we can add also to the Kinesis consumer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)