You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jiayi Liao (Jira)" <ji...@apache.org> on 2019/11/02 16:31:00 UTC
[jira] [Comment Edited] (FLINK-12675) Event time synchronization in
Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965417#comment-16965417 ]
Jiayi Liao edited comment on FLINK-12675 at 11/2/19 4:30 PM:
-------------------------------------------------------------
[~thw] Thanks for your review. I've improved some downsides of the design according to your comments. And It took me some time to look into FLIP-27 and Idle Partition related discussions. I'm concerned that this design may have conflicts with FLIP-27 because I can see FLIP-27 refactor may introduce completely new abstraction for the communication between data production (#KafkaConsumerThread) and data reads (#KafkaFetcher) in all connectors, which pretty much includes my whole design.
IMO at least we should wait until a FLIP-27 MVP or maybe I can help to consider the feedback mechanism into the new #Reader implementation. But currently there're still some works worth doing like moving #WatermarkTracker into core you mentioned (we'll finally do it either way). What do you think?
was (Author: wind_ljy):
[~thw] Thanks for your review. I've addressed some downsides of the design according to your comments. And It took me some time to look into FLIP-27 and Idle Partition related discussions. I'm concerned that this design may have conflicts with FLIP-27 because I can see FLIP-27 refactor may introduce completely new abstraction for the communication between data production (#KafkaConsumerThread) and data reads (#KafkaFetcher) in all connectors, which pretty much includes my whole design.
IMO at least we should wait until a FLIP-27 MVP or maybe I can help to consider the feedback mechanism into the new #Reader implementation. But currently there're still some works worth doing like moving #WatermarkTracker into core you mentioned (we'll finally do it either way). What do you think?
> Event time synchronization in Kafka consumer
> --------------------------------------------
>
> Key: FLINK-12675
> URL: https://issues.apache.org/jira/browse/FLINK-12675
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Kafka
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Priority: Major
>
> Integrate the source watermark tracking into the Kafka consumer and implement the sync mechanism (different consumer model, compared to Kinesis).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)