[jira] [Created] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

["Jason Gustafson (JIRA)" <ji...@apache.org>]

Jason Gustafson created KAFKA-5443:
--------------------------------------

             Summary: Consumer should use last offset from batch to set next fetch offset
                 Key: KAFKA-5443
                 URL: https://issues.apache.org/jira/browse/KAFKA-5443
             Project: Kafka
          Issue Type: Sub-task
            Reporter: Jason Gustafson
            Assignee: Jason Gustafson
             Fix For: 0.11.0.0


With message format v2, the log cleaner preserves the last offset in each batch even if the last record is removed. Currently when the batch is consumed by the consumer, we use the last record in the batch to determine the next offset to fetch. So if the last record in the batch was removed through compaction, the next fetch offset will still point to an offset in the current batch and it will be refetched. In the worst case, if the fetch size has room for that batch, the consumer will not be able to make progress. To fix this, we should advance the next fetch offset to the last offset from the batch once we have consumed that batch.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)