Best way to erase consumer state

[Ben Drees <be...@gmail.com>]

Hi,

What is the best way to erase a stopped, multiprocess consumer's state so
that it can be reset (to "smallest offsets", for example) when the consumer
is restarted?  Deleting /consumers/<group-name> from Zookeeper has had the
desired effect in the past, but I ran into a situation today where some of
the consumer processes jumped to the desired offsets (in this case
"largest") on restart but others did not.  I'm not sure whether the issue
was replication lag across the Zookeeper cluster or cached state in Kafka.
I’m using kafka-python.

Thanks,
Ben

Re: Best way to erase consumer state

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

One thing might worth checking is the offset.storage value in your
consumer config. Is it set to ³kafka² or ³zookeeper² for you consumer? If
the offset is set to kafka, then the offset will be stored in the offset
topic in Kafka instead of in zookeeper. So deleting zookeeper path won¹t
work for those consumers.
Actually the offsets for those consumers cannot be easily removed so far.
KAFKA-1784 is aiming to provide a library to make it easier.

‹Jiangjie (Becket) Qin

On 12/4/14, 10:55 PM, "Ben Drees" <be...@gmail.com> wrote:

>Hi,
>
>What is the best way to erase a stopped, multiprocess consumer's state so
>that it can be reset (to "smallest offsets", for example) when the
>consumer
>is restarted?  Deleting /consumers/<group-name> from Zookeeper has had the
>desired effect in the past, but I ran into a situation today where some of
>the consumer processes jumped to the desired offsets (in this case
>"largest") on restart but others did not.  I'm not sure whether the issue
>was replication lag across the Zookeeper cluster or cached state in Kafka.
>I¹m using kafka-python.
>
>Thanks,
>Ben