Delete Consumer Group Information

["Cuneo, Nicholas" <nc...@tycoint.com>]

We are using Kafka .9 and are playing around with the consumer group feature.  We have a lot of junk and stale consumer group information in the consumer groups and want to get rid of it.  What's the best way to do that?

Using Kafka Tool, I see that all the consumer groups are stored in 'Kafka' and not 'Zookeeper'.  When I go to zookeeper the consumers node is empty.

I tried using the ConsumerGroupCommand to delete it but since zookeeper shows no consumers this does nothing.  If the consumer groups show only being stored in Kafka, are they considered in memory and completely restarting all Kafka nodes remove them?  If so - why aren't they being replicated to zookeeper for persistence?
[cid:image003.png@01D21E37.361D9A70]
[cid:image004.png@01D21E37.361D9A70]

Thanks,

Nick Cuneo  /  Software Engineer, Cloud  /  Enterprise Software
Tel: +1 949 517 4802  /  Mobile: +1 949 243 4952
3 Ada  /  Irvine, CA 92618  /  USA
ncuneo@tycoint.com  /  www.tyco.com
[cid:image005.png@01D21E37.361D9A70]


This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected


________________________________

This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.

RE: Delete Consumer Group Information

["Krieg, David" <Da...@earlywarning.com>]

Hi Nick, 

 

First, I have the same question about removing offset groups as you do. I've
seen answers like "you don't need to" on StackOverflow, but that's obviously
not helpful. So I'll hope that part gets picked up by another user on the
group.

 

But what I can help answer is about offset storage. Using
offsets.storage=kafka in your broker configuration means that the offsets
get stored into an internal compacted topic. So, no they don't go away after
restart. 

 

The next part of your question: why aren't they being replicated to
zookeeper? This is another broker configuration property to explore for your
cluster, dual.commit.enabled=true. With this setting and
offsets.storage=kafka, the brokers will also back up to zookeeper, but this
will not allow you to delete those offsets, because changes in zookeeper
won't be applied down to kafka, it's the other way around.  

 

I see the other response that came in about expiration, and that tells how
to get offsets to expire, which will solve your "junk offset groups"
problem, as the default is they don't. But I still want to know if there's a
tool to forcibly delete a group's offsets so we can discard that group's
offset information and start fresh with the same group id, which would be
useful in a handful of use cases we've run across in switching to 0.9.

 

Seeing as how it is a compacted topic, I expect message(s) with the right
keys and a null message is the way to remove the offsets from the topic. But
is that advisable, and is there a tool to do it?

 

David Krieg | Enterprise Software Engineer

Early Warning

david.krieg@earlywarning.com 

 

From: Cuneo, Nicholas [mailto:ncuneo@tycoint.com] 
Sent: Tuesday, October 04, 2016 12:03 PM
To: users@kafka.apache.org
Subject: Delete Consumer Group Information

 

We are using Kafka .9 and are playing around with the consumer group
feature.  We have a lot of junk and stale consumer group information in the
consumer groups and want to get rid of it.  What's the best way to do that?

 

Using Kafka Tool, I see that all the consumer groups are stored in 'Kafka'
and not 'Zookeeper'.  When I go to zookeeper the consumers node is empty.

 

I tried using the ConsumerGroupCommand to delete it but since zookeeper
shows no consumers this does nothing.  If the consumer groups show only
being stored in Kafka, are they considered in memory and completely
restarting all Kafka nodes remove them?  If so - why aren't they being
replicated to zookeeper for persistence?





 

Thanks,

 

Nick Cuneo  /  Software Engineer, Cloud  /  Enterprise Software

Tel: +1 949 517 4802  /  Mobile: +1 949 243 4952
3 Ada  /  Irvine, CA 92618  /  USA

 <ma...@tycoint.com> ncuneo@tycoint.com  /   <http://www.tyco.com>
www.tyco.com



 

This email (including any attachments) may contain information that is
private or business confidential. If you received this email in error,
please delete it from your system without copying it and notify sender by
reply email so that our records can be corrected

 

 

  _____  


This e-mail contains privileged and confidential information intended for
the use of the addressees named above. If you are not the intended recipient
of this e-mail, you are hereby notified that you must not disseminate, copy
or take any action in respect of any information contained in it. If you
have received this e-mail in error, please notify the sender immediately by
e-mail and immediately destroy this e-mail and its attachments.

Re: Delete Consumer Group Information

[Vahid S Hashemian <va...@us.ibm.com>]

Please take a look at this answer to a similar recent question:
http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%3cCAHwHRrUV=M_2T_XjGwkwgZ=3BA+adOGRO1_CkVvbYTarHAGxBQ@mail.gmail.com%3e

The config parameter mentioned in the post for expiration of committed 
offsets is "offsets.retention.minutes". (see also here: 
http://stackoverflow.com/questions/39131465/how-does-an-offset-expire-for-an-apache-kafka-consumer-group
)

Regards,
--Vahid



From:   "Cuneo, Nicholas" <nc...@tycoint.com>
To:     "users@kafka.apache.org" <us...@kafka.apache.org>
Date:   10/04/2016 12:04 PM
Subject:        Delete Consumer Group Information



We are using Kafka .9 and are playing around with the consumer group 
feature.  We have a lot of junk and stale consumer group information in 
the consumer groups and want to get rid of it.  What’s the best way to do 
that?
 
Using Kafka Tool, I see that all the consumer groups are stored in ‘Kafka’ 
and not ‘Zookeeper’.  When I go to zookeeper the consumers node is empty.
 
I tried using the ConsumerGroupCommand to delete it but since zookeeper 
shows no consumers this does nothing.  If the consumer groups show only 
being stored in Kafka, are they considered in memory and completely 
restarting all Kafka nodes remove them?  If so – why aren’t they being 
replicated to zookeeper for persistence?


 
Thanks,
 
Nick Cuneo  /  Software Engineer, Cloud  /  Enterprise Software
Tel: +1 949 517 4802  /  Mobile: +1 949 243 4952
3 Ada  /  Irvine, CA 92618  /  USA
ncuneo@tycoint.com  /  www.tyco.com



This email (including any attachments) may contain information that is 
private or business confidential. If you received this email in error, 
please delete it from your system without copying it and notify sender by 
reply email so that our records can be corrected
 


This e-mail contains privileged and confidential information intended for 
the use of the addressees named above. If you are not the intended 
recipient of this e-mail, you are hereby notified that you must not 
disseminate, copy or take any action in respect of any information 
contained in it. If you have received this e-mail in error, please notify 
the sender immediately by e-mail and immediately destroy this e-mail and 
its attachments.