[jira] [Resolved] (KAFKA-15910) New group coordinator needs to generate snapshots while loading

["David Jacot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Jacot resolved KAFKA-15910.
---------------------------------
    Fix Version/s: 3.7.0
       Resolution: Fixed

> New group coordinator needs to generate snapshots while loading
> ---------------------------------------------------------------
>
>                 Key: KAFKA-15910
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15910
>             Project: Kafka
>          Issue Type: Sub-task
>            Reporter: Jeff Kim
>            Assignee: Jeff Kim
>            Priority: Major
>             Fix For: 3.7.0
>
>
> After the new coordinator loads a __consumer_offsets partition, it logs the following exception when making a read operation (fetch/list groups, etc):
>  
> {{{}java.lang.RuntimeException: No in-memory snapshot for epoch 740745. Snapshot epochs are:{}}}{{{}at org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:178){}}}{{{}at org.apache.kafka.timeline.SnapshottableHashTable.snapshottableIterator(SnapshottableHashTable.java:407){}}}{{{}at org.apache.kafka.timeline.TimelineHashMap$ValueIterator.<init>(TimelineHashMap.java:283){}}}{{{}at org.apache.kafka.timeline.TimelineHashMap$Values.iterator(TimelineHashMap.java:271){}}}
> {{...}}
>  
> This happens because we don't have a snapshot at the last updated high watermark after loading. We cannot generate a snapshot at the high watermark after loading all batches because it may contain records that have not yet been committed. We also don't know where the high watermark will advance up to so we need to generate a snapshot for each offset the loader observes to be greater than the current high watermark. Then once we add the high watermark listener and update the high watermark we can delete all of the snapshots prior. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)