Offsets of deleted consumer groups do not get deleted correctly

[Claudia Wegmann <c....@kasasi.de>]

Hey there,

I've got the problem that the "__consumer_offsets" topic grows pretty big over time. After some digging, I found offsets for consumer groups that were deleted a long time ago still being present in the topic. Many of them are offsets for console consumers, that have been deleted with "kafka-consumer-groups.sh --delete --group ...".

As far as I understand log cleaning, those offsets should have been deleted a long time ago, because these consumers are no longer active. When I query "kafka-consumer-groups.sh --bootstrap-server ...  --list" I don't see those consumers either.

Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that let's kafka hang on to those consumer groups?

How can I get the log cleaner to delete these old offsets? Is there another way than setting "cleanup.policy" to "delete"?

Thanks for our help!

Best,
Claudia

Re: Offsets of deleted consumer groups do not get deleted correctly

[Patrik Kleindl <pk...@gmail.com>]

Hi Claudia
Just a sidenote, there is a combined policy for "compact, delete" which
deletes messages older than retention.ms and compacts newer ones if I
remember correctly.
It's still not really in the docs as it seems
https://kafka.apache.org/documentation/#topicconfigs
best regards
Patrik

On Mon, 1 Apr 2019 at 10:02, Claudia Wegmann <c....@kasasi.de> wrote:

> Hi,
>
> thanks for your reply.
>
> Also the groups were deleted month ago, there are still valid values
> there. So I guess deleting the group did not produce the tombstone record
> correctly. Your explanation made it clearer for me. I know I should keep
> cleanup policy compact in general. I decided to switch to policy for just
> long enough, so that these old records get deleted. I guess I could have
> produced the tombstone recods to be on the safe side.
>
> Thanks,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Vincent Maurin <vi...@gmail.com>
> Gesendet: Freitag, 29. März 2019 15:24
> An: users@kafka.apache.org
> Betreff: Re: Offsets of deleted consumer groups do not get deleted
> correctly
>
> Hi,
>
> You should keep the policy compact for the topic __consumer_offsets This
> topic stores for each group/topic/partition the offset consumed. As only
> the latest message for a group/topic/partition is relevant, the policy
> compact will keep only this message. When you delete a group, actually it
> will produce a tombstone to this topic (i.e body NULL). Then when the log
> compaction is running, it will definitively remove the tombstone.
> But to have an effective delete of the tombstones, keep in mind :
> * compaction runs only on rolled out segments
> * deletion of tombstone only occurs if the delete.retention.ms delay is
> expired
>
> Best regards
>
> On Fri, Mar 29, 2019 at 2:16 PM Claudia Wegmann <c....@kasasi.de>
> wrote:
>
> > Hey there,
> >
> > I've got the problem that the "__consumer_offsets" topic grows pretty
> > big over time. After some digging, I found offsets for consumer groups
> > that were deleted a long time ago still being present in the topic.
> > Many of them are offsets for console consumers, that have been deleted
> > with "kafka-consumer-groups.sh --delete --group ...".
> >
> > As far as I understand log cleaning, those offsets should have been
> > deleted a long time ago, because these consumers are no longer active.
> > When I query "kafka-consumer-groups.sh --bootstrap-server ...  --list"
> > I don't see those consumers either.
> >
> > Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that
> > let's kafka hang on to those consumer groups?
> >
> > How can I get the log cleaner to delete these old offsets? Is there
> > another way than setting "cleanup.policy" to "delete"?
> >
> > Thanks for our help!
> >
> > Best,
> > Claudia
> >
>

AW: Offsets of deleted consumer groups do not get deleted correctly

[Claudia Wegmann <c....@kasasi.de>]

Hi,

thanks for your reply.

Also the groups were deleted month ago, there are still valid values there. So I guess deleting the group did not produce the tombstone record correctly. Your explanation made it clearer for me. I know I should keep cleanup policy compact in general. I decided to switch to policy for just long enough, so that these old records get deleted. I guess I could have produced the tombstone recods to be on the safe side.

Thanks,
Claudia

-----Ursprüngliche Nachricht-----
Von: Vincent Maurin <vi...@gmail.com> 
Gesendet: Freitag, 29. März 2019 15:24
An: users@kafka.apache.org
Betreff: Re: Offsets of deleted consumer groups do not get deleted correctly

Hi,

You should keep the policy compact for the topic __consumer_offsets This topic stores for each group/topic/partition the offset consumed. As only the latest message for a group/topic/partition is relevant, the policy compact will keep only this message. When you delete a group, actually it will produce a tombstone to this topic (i.e body NULL). Then when the log compaction is running, it will definitively remove the tombstone.
But to have an effective delete of the tombstones, keep in mind :
* compaction runs only on rolled out segments
* deletion of tombstone only occurs if the delete.retention.ms delay is expired

Best regards

On Fri, Mar 29, 2019 at 2:16 PM Claudia Wegmann <c....@kasasi.de> wrote:

> Hey there,
>
> I've got the problem that the "__consumer_offsets" topic grows pretty 
> big over time. After some digging, I found offsets for consumer groups 
> that were deleted a long time ago still being present in the topic. 
> Many of them are offsets for console consumers, that have been deleted 
> with "kafka-consumer-groups.sh --delete --group ...".
>
> As far as I understand log cleaning, those offsets should have been 
> deleted a long time ago, because these consumers are no longer active. 
> When I query "kafka-consumer-groups.sh --bootstrap-server ...  --list" 
> I don't see those consumers either.
>
> Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that 
> let's kafka hang on to those consumer groups?
>
> How can I get the log cleaner to delete these old offsets? Is there 
> another way than setting "cleanup.policy" to "delete"?
>
> Thanks for our help!
>
> Best,
> Claudia
>

Re: Offsets of deleted consumer groups do not get deleted correctly

[Vincent Maurin <vi...@gmail.com>]

Hi,

You should keep the policy compact for the topic __consumer_offsets
This topic stores for each group/topic/partition the offset consumed. As
only the latest message for a group/topic/partition is relevant, the policy
compact will keep only this message. When you delete a group, actually it
will produce a tombstone to this topic (i.e body NULL). Then when the log
compaction is running, it will definitively remove the tombstone.
But to have an effective delete of the tombstones, keep in mind :
* compaction runs only on rolled out segments
* deletion of tombstone only occurs if the delete.retention.ms delay is
expired

Best regards

On Fri, Mar 29, 2019 at 2:16 PM Claudia Wegmann <c....@kasasi.de> wrote:

> Hey there,
>
> I've got the problem that the "__consumer_offsets" topic grows pretty big
> over time. After some digging, I found offsets for consumer groups that
> were deleted a long time ago still being present in the topic. Many of them
> are offsets for console consumers, that have been deleted with
> "kafka-consumer-groups.sh --delete --group ...".
>
> As far as I understand log cleaning, those offsets should have been
> deleted a long time ago, because these consumers are no longer active. When
> I query "kafka-consumer-groups.sh --bootstrap-server ...  --list" I don't
> see those consumers either.
>
> Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that
> let's kafka hang on to those consumer groups?
>
> How can I get the log cleaner to delete these old offsets? Is there
> another way than setting "cleanup.policy" to "delete"?
>
> Thanks for our help!
>
> Best,
> Claudia
>