[jira] [Resolved] (FLINK-3375) Allow Watermark Generation in the Kafka Source

["Stephan Ewen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephan Ewen resolved FLINK-3375.
---------------------------------
       Resolution: Fixed
    Fix Version/s:     (was: 1.0.0)
                   1.1.0

Finalized in 2dcd27f403c2a7f10791bfe21c45e2a326aa46a1

> Allow Watermark Generation in the Kafka Source
> ----------------------------------------------
>
>                 Key: FLINK-3375
>                 URL: https://issues.apache.org/jira/browse/FLINK-3375
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>    Affects Versions: 1.0.0
>            Reporter: Stephan Ewen
>            Assignee: Kostas Kloudas
>             Fix For: 1.1.0
>
>
> It is a common case that event timestamps are ascending inside one Kafka Partition. Ascending timestamps are easy for users, because they are handles by ascending timestamp extraction.
> If the Kafka source has multiple partitions per source task, then the records become out of order before timestamps can be extracted and watermarks can be generated.
> If we make the FlinkKafkaConsumer an event time source function, it can generate watermarks itself. It would internally implement the same logic as the regular operators that merge streams, keeping track of event time progress per partition and generating watermarks based on the current guaranteed event time progress.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)