You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "David Jacot (Jira)" <ji...@apache.org> on 2023/06/10 16:29:00 UTC

[jira] [Assigned] (KAFKA-15079) Offset expirations in topics with less event generation results in messages that take months to be delivered.

     [ https://issues.apache.org/jira/browse/KAFKA-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Jacot reassigned KAFKA-15079:
-----------------------------------

    Assignee: Ritika Muduganti

> Offset expirations in topics with less event generation results in messages that take months to be delivered.
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-15079
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15079
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator
>            Reporter: Ritika Muduganti
>            Assignee: Ritika Muduganti
>            Priority: Major
>
> It was observed that messages were taking months to be delivered in certain pipelines. The offsets had been reset and consumption restarted from messages that were a couple months old.
> On further investigation the team found that once the offsets.retention.minutes (7 days) elapses and there's no subscribers to the topic, the offsets are subject to immediate expiration. This can be a problem when we close the current consumer and the retention period has already passed due to the infrequent nature of offset commits. The offsets then expire before a new consumer can come up and subscribe to the topic again. Since only one consumer could be subscribed to the topic it will be removed from the subscription list as soon as the consumer leaves. 
> Two things to keep in mind here are :-
>  # The consumer group has different consumers subscribed to different topics.
>  # Events aren't generated very frequently to these topics.
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadata.scala#LL766C1-L825C4]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)