[jira] [Commented] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

["Jaya Ananthram (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101035#comment-17101035 ] 

Jaya Ananthram commented on FLINK-5479:
---------------------------------------

Hi All - Is there is any temporary work around regarding this ticket? I see that FLINK-10740 is in progress but not sure when it will available for release, in the mean time if any one able to share any work around for this, that would be great.

> Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-5479
>                 URL: https://issues.apache.org/jira/browse/FLINK-5479
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>            Reporter: Tzu-Li (Gordon) Tai
>            Priority: Major
>
> Reported in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-topic-partition-skewness-causes-watermark-not-being-emitted-td11008.html
> Similar to what's happening to idle sources blocking watermark progression in downstream operators (see FLINK-5017), the per-partition watermark mechanism in {{FlinkKafkaConsumer}} is also being blocked of progressing watermarks when a partition is idle. The watermark of idle partitions is always {{Long.MIN_VALUE}}, therefore the overall min watermark across all partitions of a consumer subtask will never proceed.
> It's normally not a common case to have Kafka partitions not producing any data, but it'll probably be good to handle this as well. I think we should have a localized solution similar to FLINK-5017 for the per-partition watermarks in {{AbstractFetcher}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)