Consumer keeps committing to a partition it no longer holds after a rebalance

[Saurabh Daftary <sa...@gmail.com>]

I am running broker 0.9.0 with consumer 0.9.0.1.

Seeing an issue wherein after a consumer rebalance (when I add a new
consumer) my old consumer still keeps committing offsets to partition it no
longer holds after a rebalance.

The rebalance seem to work ok - On run ConsumerGroupCommand I see that the
partitions are redistributed after I add a new consumer. The respective
consumers do their fetches only on their assigned partitions but the old
consumer still commits the same last offset it had when it was holding on
to that partition At the same time the added consumer commits the correct
offsets.

Any inputs on what could be going on?

Re: Consumer keeps committing to a partition it no longer holds after a rebalance

[Saurabh Daftary <sa...@gmail.com>]

Yes I can reliably reproduce this.

To reproduce it initially start with a single consumer node so that it owns
all the partitions. Then add another consumer node, after rebalancing you
would see that although the ownership of certain partitions are with the
new node the older node which now only owns a subsets of partitions is
still committing to the partitions that are not owned by it.

This lasts for atleast 30-40 minutes at times even longer.

On Wed, Mar 23, 2016 at 6:29 AM, Saurabh Daftary <sa...@gmail.com>
wrote:

> I am running broker 0.9.0 with consumer 0.9.0.1.
>
> Seeing an issue wherein after a consumer rebalance (when I add a new
> consumer) my old consumer still keeps committing offsets to partition it no
> longer holds after a rebalance.
>
> The rebalance seem to work ok - On run ConsumerGroupCommand I see that the
> partitions are redistributed after I add a new consumer. The respective
> consumers do their fetches only on their assigned partitions but the old
> consumer still commits the same last offset it had when it was holding on
> to that partition At the same time the added consumer commits the correct
> offsets.
>
> Any inputs on what could be going on?
>

Re: Consumer keeps committing to a partition it no longer holds after a rebalance

[Christian Posta <ch...@gmail.com>]

How long does this last? Maybe there are some async commit-offset frames
left in the send buffer to that node? Can you reliably reproduce this?

On Wed, Mar 23, 2016 at 6:29 AM, Saurabh Daftary <sa...@gmail.com>
wrote:

> I am running broker 0.9.0 with consumer 0.9.0.1.
>
> Seeing an issue wherein after a consumer rebalance (when I add a new
> consumer) my old consumer still keeps committing offsets to partition it no
> longer holds after a rebalance.
>
> The rebalance seem to work ok - On run ConsumerGroupCommand I see that the
> partitions are redistributed after I add a new consumer. The respective
> consumers do their fetches only on their assigned partitions but the old
> consumer still commits the same last offset it had when it was holding on
> to that partition At the same time the added consumer commits the correct
> offsets.
>
> Any inputs on what could be going on?
>



-- 
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io