You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@samza.apache.org by "Jake Maes (JIRA)" <ji...@apache.org> on 2017/11/01 20:30:00 UTC

[jira] [Created] (SAMZA-1481) KafkaSystemConsumer does not properly track highwatermark in some cases.

Jake Maes created SAMZA-1481:
--------------------------------

             Summary: KafkaSystemConsumer does not properly track highwatermark in some cases. 
                 Key: SAMZA-1481
                 URL: https://issues.apache.org/jira/browse/SAMZA-1481
             Project: Samza
          Issue Type: Bug
            Reporter: Jake Maes
            Assignee: Jake Maes
             Fix For: 0.14.0


The KafkaSystemConsumer currently only updates the isAtHighWatermark when it receives a payload from Kafka. 

This does not work when:
1. The topic is empty
2. The oldest or newest offset changes on the broker
3. An invalid initial offset is used for the consumer and the Kafka auto reset policy kicks in. 

The isAtHighWatermark flag is crucial for topics that we bootstrap (checkpoint, changelog, coordinator) because we poll infinitely until we're at head of stream. And if this flag is wrong, we poll indefinitely.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)