You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/06/14 04:23:00 UTC

[jira] [Commented] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

    [ https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048688#comment-16048688 ] 

ASF GitHub Bot commented on KAFKA-5443:
---------------------------------------

GitHub user hachikuji opened a pull request:

    https://github.com/apache/kafka/pull/3331

    KAFKA-5443: Consumer should use last offset from batch to set next fetch offset

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/hachikuji/kafka KAFKA-5443

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/3331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3331
    
----
commit b991c8185c5bce1cedb39dc5158afaed1b8db2f4
Author: Jason Gustafson <ja...@confluent.io>
Date:   2017-06-14T04:17:33Z

    KAFKA-5443: Consumer should use last offset from batch to set next fetch offset

----


> Consumer should use last offset from batch to set next fetch offset
> -------------------------------------------------------------------
>
>                 Key: KAFKA-5443
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5443
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: clients, core, producer 
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>             Fix For: 0.11.0.0
>
>
> With message format v2, the log cleaner preserves the last offset in each batch even if the last record is removed. Currently when the batch is consumed by the consumer, we use the last record in the batch to determine the next offset to fetch. So if the last record in the batch was removed through compaction, the next fetch offset will still point to an offset in the current batch and it will be refetched. In the worst case, if the fetch size has room for that batch, the consumer will not be able to make progress. To fix this, we should advance the next fetch offset to the last offset from the batch once we have consumed that batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)