Offset not written back to zookeeper

[Raimar Falke <ra...@optivo.de>]

Hello

We are using a modified version of kafka 0.7.1 (https://github.com/optivo-org/kafka)
and had at least one case where the consumers of the group got events but the offsets
were not written back to zookeeper. The consumers stopped updating the offsets after
a release where the consumers of the group are each individually is restarted. Here 
a lot of rebalancing was taking place. After a further restart of all consumers the 
events were processed a second time. No related warning or error message could be 
found.

Is this a known problem? What could be the cause for this?

Tracing the code paths it seems weird, that ZookeeperConsumerConnector.commitOffsets() 
does not issue a warning if no offsets were written. Is it possible that a rebalancing
was not finished and as such events were processed but without an ownership no updates
take place?

    Raimar
 
optivo GmbH
Wallstraße 16
10179 Berlin 
 
Telefon: 030 / 76 80 78 163
Fax:     030 / 76 80 78 499
 
E-Mail:  raimar.falke@optivo.de
Website: http://www.optivo.de
    
Handelsregister: HRB Berlin 88738
Geschäftsführer: Peter Romianowski, Ulf Richter

Re: Offset not written back to zookeeper

[Jun Rao <ju...@gmail.com>]

Normally, the offset commit thread won't die since we catch and log a
warning if we hit any zk exception. If this happens again, you can also use
the VerifyConsumerRebalance tool to see if rebalance has completed
successfully.

Thanks,

Jun

On Tue, Oct 30, 2012 at 2:51 AM, Raimar Falke <ra...@optivo.de>wrote:

> > How is that modified version different from 0.7.1?
>
> Multiple small changes. Some from the Kafka-Jira some self rolled:
> https://github.com/optivo-
> org/kafka/commits/trunk
>
> > Did you take a thread dump and see what the offset commit thread was
> doing?
>
> Sadly no.
>
> > > Tracing the code paths it seems weird, that
> > > ZookeeperConsumerConnector.commitOffsets()
> > > does not issue a warning if no offsets were written.
>
> Can this happen in a normal working condition that no warning is issues?
>
>    Raimar
>

Re: Offset not written back to zookeeper

[Jun Rao <ju...@gmail.com>]

How is that modified version different from 0.7.1? Did you take a thread
dump and see what the offset commit thread was doing?

Thanks,

Jun

On Mon, Oct 29, 2012 at 7:34 AM, Raimar Falke <ra...@optivo.de>wrote:

> Hello
>
> We are using a modified version of kafka 0.7.1 (
> https://github.com/optivo-org/kafka)
> and had at least one case where the consumers of the group got events but
> the offsets
> were not written back to zookeeper. The consumers stopped updating the
> offsets after
> a release where the consumers of the group are each individually is
> restarted. Here
> a lot of rebalancing was taking place. After a further restart of all
> consumers the
> events were processed a second time. No related warning or error message
> could be
> found.
>
> Is this a known problem? What could be the cause for this?
>
> Tracing the code paths it seems weird, that
> ZookeeperConsumerConnector.commitOffsets()
> does not issue a warning if no offsets were written. Is it possible that a
> rebalancing
> was not finished and as such events were processed but without an
> ownership no updates
> take place?
>
>     Raimar
>
> optivo GmbH
> Wallstraße 16
> 10179 Berlin
>
> Telefon: 030 / 76 80 78 163
> Fax:     030 / 76 80 78 499
>
> E-Mail:  raimar.falke@optivo.de
> Website: http://www.optivo.de
>
> Handelsregister: HRB Berlin 88738
> Geschäftsführer: Peter Romianowski, Ulf Richter
>