Rebalance events when auto commit is disabled in 0.9 consumer

[Balthasar Schopman <b....@tech.leaseweb.com>]

Hi,

Per consumer group we have many consumers that aggregate data for a predefined amount of time and then write the aggregated data to a db. We're looking to set enable.auto.commit to false and write the offsets only when the aggregated data has been written to db. In that context we're not sure what happens when a rebalance occurs.

The rebalance can occur at any point in time, e.g. when a new consumer joins the party. What will happen with the existing consumer at such a moment? It is likely that they have consumed messages from the partitions that they were owning and suddenly they will own different partitions.

What does the Kafka 0.9 client do if we don't provide a ConsumerRebalanceListener?

Will we have to implement a ConsumerRebalanceListener to act on such an event?

Kind regards,

Balthasar Schopman
DevOps Software Engineer
LeaseWeb Technologies B.V.

T: +31 20 316 0232
M:
E: b.schopman@tech.leaseweb.com
W: http://www.leaseweb.com

Luttenbergweg 8, 1101 EC Amsterdam, Netherlands

Re: Rebalance events when auto commit is disabled in 0.9 consumer

[Liquan Pei <li...@gmail.com>]

Hi Balthasar,

It is possible that after rebalance, the topic partitions assigned to a
consumer instance are changed. To prevent state loss, a custom
ConsumerRebalanceListener is needed to properly save the state. In the
onPartitionRevoked method, you need to trigger the write to database and
wait for the write to complete and then invoke synchronous offset commit.
Also, you should make sure that the time spent on these operations does not
exceed the session.timout.ms.

Thanks,
Liquan

On Tue, Apr 26, 2016 at 2:11 AM, Balthasar Schopman <
b.schopman@tech.leaseweb.com> wrote:

> Hi,
>
> Per consumer group we have many consumers that aggregate data for a
> predefined amount of time and then write the aggregated data to a db. We're
> looking to set enable.auto.commit to false and write the offsets only when
> the aggregated data has been written to db. In that context we're not sure
> what happens when a rebalance occurs.
>
> The rebalance can occur at any point in time, e.g. when a new consumer
> joins the party. What will happen with the existing consumer at such a
> moment? It is likely that they have consumed messages from the partitions
> that they were owning and suddenly they will own different partitions.
>
> What does the Kafka 0.9 client do if we don't provide a
> ConsumerRebalanceListener?
>
> Will we have to implement a ConsumerRebalanceListener to act on such an
> event?
>
> Kind regards,
>
> Balthasar Schopman
> DevOps Software Engineer
> LeaseWeb Technologies B.V.
>
> T: +31 20 316 0232
> M:
> E: b.schopman@tech.leaseweb.com
> W: http://www.leaseweb.com
>
> Luttenbergweg 8, 1101 EC Amsterdam, Netherlands
>
>
>


-- 
Liquan Pei
Software Engineer, Confluent Inc