Question about offset expiration

[Mat Arye <ma...@iobeam.com>]

I have a question about the config value offsets.retention.minutes in kafka
0.9.0.0.
Is this the timeout for when offsets get compacted in the topic or actually
deleted (as it appears from a cursory reading of
https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/coordinator/GroupMetadataManager.scala#L532
)

If it actually deletes the offsets, what happens if a topic simply doesn't
get a new message for a day (the default value for
offsets.retention.minutes). Are the offsets then forgotten? This seems
really strange and counter-intuitive. I thought offsets were supposed to be
saved forever (isn't that why a compacted topic is used)?

Thanks in advance,
Matvey Arye

Re: Question about offset expiration

[Joel Koshy <jj...@gmail.com>]

Hi Matvey,


I have a question about the config value offsets.retention.minutes in kafka
> 0.9.0.0.
> Is this the timeout for when offsets get compacted in the topic or actually
> deleted (as it appears from a cursory reading of
>
> https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/coordinator/GroupMetadataManager.scala#L532
> )
>

It is actually deleted at that point. i.e., it is removed from the
in-memory cache and we append a tombstone for it in the log (which means it
will get deleted whenever that segment rolls over and becomes eligible for
compaction). So a subsequent offset fetch will return no offset.


> If it actually deletes the offsets, what happens if a topic simply doesn't
> get a new message for a day (the default value for
> offsets.retention.minutes). Are the offsets then forgotten? This seems
>

Yes


> really strange and counter-intuitive. I thought offsets were supposed to be
> saved forever (isn't that why a compacted topic is used)?
>

One reason to purge offsets older than the configured retention is to
clean-up after consumers that are no longer active (especially short-lived
consumers such as console consumers). Users can certainly opt to override
this behavior by setting it to a very high value. (Although it sounds like
you are suggesting that "forever" should be the default in the first place?)

Joel