[jira] [Updated] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

["Ismael Juma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismael Juma updated KAFKA-5443:
-------------------------------
    Status: Patch Available  (was: Open)

> Consumer should use last offset from batch to set next fetch offset
> -------------------------------------------------------------------
>
>                 Key: KAFKA-5443
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5443
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: clients, core, producer 
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>             Fix For: 0.11.0.0
>
>
> With message format v2, the log cleaner preserves the last offset in each batch even if the last record is removed. Currently when the batch is consumed by the consumer, we use the last record in the batch to determine the next offset to fetch. So if the last record in the batch was removed through compaction, the next fetch offset will still point to an offset in the current batch and it will be refetched. In the worst case, if the fetch size has room for that batch, the consumer will not be able to make progress. To fix this, we should advance the next fetch offset to the last offset from the batch once we have consumed that batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)