You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Stephan Ewen (JIRA)" <ji...@apache.org> on 2016/09/27 19:08:20 UTC

[jira] [Created] (FLINK-4702) Kafka consumer must commit offsets asynchronously

Stephan Ewen created FLINK-4702:
-----------------------------------

             Summary: Kafka consumer must commit offsets asynchronously
                 Key: FLINK-4702
                 URL: https://issues.apache.org/jira/browse/FLINK-4702
             Project: Flink
          Issue Type: Bug
          Components: Kafka Connector
    Affects Versions: 1.1.2
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen
            Priority: Blocker
             Fix For: 1.2.0, 1.1.3


The offset commit calls to Kafka may occasionally take very long.
In that case, the {{notifyCheckpointComplete()}} method blocks for long and the KafkaConsumer cannot make progress and cannot perform checkpoints.

Kafka 0.9+ have methods to commit asynchronously.
We should use those and make sure no more than one commit is concurrently in progress, to that commit requests do not pile up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)