Auto offset commit failed

["yuanjia8947@163.com" <yu...@163.com>]

Hi All,
    My kafka cluster is kafka0.10.0.
    I have found two reasons for Auto offset commit failed in the log file. One is Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member, and another is Commit offsets failed with retriable exception. You should retry committing offsets.
    My questions are:
   1. When the consumer rejoin group?
   2. Could it be consuming the message repeatly during rebalancing?
  
    Thanks.



Yuanjia Li

Re: Auto offset commit failed

[Guozhang Wang <wa...@gmail.com>]

Hello Yuanjia,

1) You can take a look at this wiki for details in consumer (re)-join group.

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal

In short summary, the coordinator will detect if a consumer has failed
based on the heartbeat protocol, and if yes tells other consumers to
re-join the group in order to rebalance.

2) That is possible, if the consumer who is owning the partition does not
commit the offsets before revoking the assigned partition, for example if
it was kicked out of the group by the coordinator (in the above scenario),
then the newly assigned consumer of the partition will resume from the
previously committed offsets, and hence some of the consumed records from
the previous owner maybe re-consumed again.

Guozhang


On Tue, Aug 30, 2016 at 1:26 AM, yuanjia8947@163.com <yu...@163.com>
wrote:

> Hi All,
>     My kafka cluster is kafka0.10.0.
>     I have found two reasons for Auto offset commit failed in the log
> file. One is Commit cannot be completed since the group has already
> rebalanced and assigned the partitions to another member, and another is
> Commit offsets failed with retriable exception. You should retry committing
> offsets.
>     My questions are:
>    1. When the consumer rejoin group?
>    2. Could it be consuming the message repeatly during rebalancing?
>
>     Thanks.
>
>
>
> Yuanjia Li
>



-- 
-- Guozhang