[jira] [Commented] (KAFKA-1006) Consumer loses messages of a new topic with auto.offset.reset = largest

["Patrick Te Tau (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472427#comment-15472427 ] 

Patrick Te Tau commented on KAFKA-1006:
---------------------------------------

Hi [~guozhang], we are also having trouble with our integration tests. 
I can manually set the offset for new topics but this will break my subscription to pre-existing topics. Because have no way of telling whether the topic is a new one or an old one, I have no way to switch my strategy.
I have considered storing a list of topics on my client but this solution fails when I run multiple clients. 
Any suggestions?


> Consumer loses messages of a new topic with auto.offset.reset = largest
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-1006
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1006
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.0
>            Reporter: Swapnil Ghike
>            Assignee: Guozhang Wang
>              Labels: usability
>
> Consumer currently uses auto.offset.reset = largest by default. If a new topic is created, consumer's topic watcher is fired. The consumer will first finish partition reassignment as part of rebalance and then start consuming from the tail of each partition. Until the partition reassignment is over, the server may have appended new messages to the new topic, consumer won't consume these messages. Thus, multiple batches of messages may be lost when a topic is newly created. 
> The fix is to start consuming from the earliest offset for newly created topics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)