[jira] [Commented] (KAFKA-346) Don't call commitOffsets() during rebalance

["Peter Romianowski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278850#comment-13278850 ] 

Peter Romianowski commented on KAFKA-346:
-----------------------------------------

I could now write a patch that - as Jun suggested - does not commit offsets, if autoCommit has been turned off. Together with KAFKA-345 that adds a listener for rebalance-events, we could simply use a listener and throw away all messages in our batch upon a rebalance. That would work fine for us.

My question is: Do you think this is the behavior others would expect? Even though the current behavior is broken IMHO, the new one would introduce quite some duplicate messages which have to be handled by the application.

Are there better ideas on how to fix this?
                
> Don't call commitOffsets() during rebalance
> -------------------------------------------
>
>                 Key: KAFKA-346
>                 URL: https://issues.apache.org/jira/browse/KAFKA-346
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.7, 0.8, 0.7.1
>            Reporter: Peter Romianowski
>
> A sample use-case
> If I read the source correctly then offsets can be committed at any
> time (whenever there is a change in consumer or broker zk registry).
> Our application doesn't use auto-commit in order to batch some
> messages together, process them and if everything went fine, we call
> commitOffsets(). If, for any reason, the processing of messages does
> not succeed, we rely on Kafka's promise to re-deliver the messages.
> But if ZKRebalancerListener triggers a rebalance before our "batch" of
> messages is full, then offsets will be committed even if the messages
> have not been processed yet by our application. So if then processing
> of these messages fails, we basically lost them, right?
> (See discussion at http://www.mail-archive.com/kafka-users@incubator.apache.org/msg01415.html)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira