ZookeeperConsumerConnector calls commitOffsets on rebalance

[Peter Romianowski <ho...@googlemail.com>]

Hi,

should have found a better subject...

If I read the source correctly then offsets can be committed at any
time (whenever there is a change in consumer or broker zk registry).
Our application doesn't use auto-commit in order to batch some
messages together, process them and if everything went fine, we call
commitOffsets(). If, for any reason, the processing of messages does
not succeed, we rely on Kafka's promise to re-deliver the messages.

But if ZKRebalancerListener triggers a rebalance before our "batch" of
messages is full, then offsets will be committed even if the messages
have not been processed yet by our application. So if then processing
of these messages fails, we basically lost them, right?

Are there any recipes how to handle this? I really would like to stick
to ZookeeperConsumerConnector, so switching to SimpleConsumer would be
considered as "last resort".

Regards

Peter

--
404 Signature Not Found

Re: ZookeeperConsumerConnector calls commitOffsets on rebalance

[Peter Romianowski <ho...@googlemail.com>]

Hi Jun,

I created KAFKA-346 and added some comments to it (to move discussion to JIRA).

Greetings

Peter


2012/5/17 Jun Rao <ju...@gmail.com>:
> Peter,
>
> We probably shouldn't commit offsets automatically during rebalance, if
> auto offset commit is turned off. The caveat is that during a rebalance,
> all message since the last manual commit point will be redelivered to the
> consumer. Could you create a jira and maybe submit a patch for this?
>
> Thanks,
>
> Jun
>
> On Thu, May 17, 2012 at 8:12 AM, Peter Romianowski
> <ho...@googlemail.com>wrote:
>
>> Hi,
>>
>> should have found a better subject...
>>
>> If I read the source correctly then offsets can be committed at any
>> time (whenever there is a change in consumer or broker zk registry).
>> Our application doesn't use auto-commit in order to batch some
>> messages together, process them and if everything went fine, we call
>> commitOffsets(). If, for any reason, the processing of messages does
>> not succeed, we rely on Kafka's promise to re-deliver the messages.
>>
>> But if ZKRebalancerListener triggers a rebalance before our "batch" of
>> messages is full, then offsets will be committed even if the messages
>> have not been processed yet by our application. So if then processing
>> of these messages fails, we basically lost them, right?
>>
>> Are there any recipes how to handle this? I really would like to stick
>> to ZookeeperConsumerConnector, so switching to SimpleConsumer would be
>> considered as "last resort".
>>
>> Regards
>>
>> Peter
>>
>> --
>> 404 Signature Not Found
>>



-- 
---
404 Signature Not Found

Re: ZookeeperConsumerConnector calls commitOffsets on rebalance

[Jun Rao <ju...@gmail.com>]

Peter,

We probably shouldn't commit offsets automatically during rebalance, if
auto offset commit is turned off. The caveat is that during a rebalance,
all message since the last manual commit point will be redelivered to the
consumer. Could you create a jira and maybe submit a patch for this?

Thanks,

Jun

On Thu, May 17, 2012 at 8:12 AM, Peter Romianowski
<ho...@googlemail.com>wrote:

> Hi,
>
> should have found a better subject...
>
> If I read the source correctly then offsets can be committed at any
> time (whenever there is a change in consumer or broker zk registry).
> Our application doesn't use auto-commit in order to batch some
> messages together, process them and if everything went fine, we call
> commitOffsets(). If, for any reason, the processing of messages does
> not succeed, we rely on Kafka's promise to re-deliver the messages.
>
> But if ZKRebalancerListener triggers a rebalance before our "batch" of
> messages is full, then offsets will be committed even if the messages
> have not been processed yet by our application. So if then processing
> of these messages fails, we basically lost them, right?
>
> Are there any recipes how to handle this? I really would like to stick
> to ZookeeperConsumerConnector, so switching to SimpleConsumer would be
> considered as "last resort".
>
> Regards
>
> Peter
>
> --
> 404 Signature Not Found
>