[jira] [Updated] (KAFKA-6209) Consumer lag is inconsistent when manually committing offset for transactionnal messages

["Fabien Chaillou (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fabien Chaillou updated KAFKA-6209:
-----------------------------------
    Summary: Consumer lag is inconsistent when manually committing offset for transactionnal messages  (was: Lag is inconsistent when manually committing offset for transactionnal messages)

> Consumer lag is inconsistent when manually committing offset for transactionnal messages
> ----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6209
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6209
>             Project: Kafka
>          Issue Type: Improvement
>          Components: consumer
>    Affects Versions: 0.11.0.1
>            Reporter: Fabien Chaillou
>
> Hi,
> I'm using kafka consumers paired with kafka producers to have transactionnal consume -> produce -> commit flow.
> I checked the kafka-consumer-groups.sh tool to make sure my consumer were consuming all the messages and it appears that my consumer always have a lag of 1.
> After more digging, I discovered that kafka uses Control records for transaction handling and those records are not returned to the consumer. So if the last record published in the topic is part of a transaction then kafka will add a control record after it and the client code will only be able to send a commit request up to the last control record and the lag will be computed as 1.
> It is not really a bug per se but an inconsistent behavior and I had to dig into the exactly-once KIP (KIP-98) and the consumer code to figure this out.
> I think the issue should at least be documented somewhere as I'm honestly not sure of the proper fix. 
> Thanks for your feedback.
> Fabien



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)