[jira] [Created] (KAFKA-19862) Group coordinator loading may fail when there is concurrent compaction

["Sean Quah (Jira)" <ji...@apache.org>]

Sean Quah created KAFKA-19862:
---------------------------------

             Summary: Group coordinator loading may fail when there is concurrent compaction
                 Key: KAFKA-19862
                 URL: https://issues.apache.org/jira/browse/KAFKA-19862
             Project: Kafka
          Issue Type: Bug
          Components: group-coordinator
            Reporter: Sean Quah
            Assignee: Sean Quah


For consumer and streams groups, we reject replay of {{Consumer/StreamsGroupCurrentMemberAssignment}} records when we detect a partition / task is already owned by another member.

During group coordinator load, we replay the records in {{{}__consumer_offsets{}}}. When compaction is running concurrently, we can load uncompacted data, followed by a newly swapped in compacted segment, followed by the uncompacted head of the log. This allows for situations where the record unassigning a partition/task is missed during loading.

eg.
We can load a record \{ Member A is assigned partition X },
then miss the record \{ Member A is unassigned partition X },
then load the record \{ Member B is assigned partition X }, which fails with an exception like

{{[GroupCoordinator id=2] Failed to load metadata from __consumer_offsets-4 with epoch 10 due to java.lang.RuntimeException: Replaying record CoordinatorRecord(key=ConsumerGroupCurrentMemberAssignmentKey(groupId='...', memberId='ZxHk7W53S_aHFdpxYc-_Jw'), value=ApiMessageAndVersion(ConsumerGroupCurrentMemberAssignmentValue(memberEpoch=854659, previousMemberEpoch=854633, state=0, assignedPartitions=[TopicPartitions(topicId=9lL1aTMuSC22QAXsHgzhew, partitions=[1, 2]), TopicPartitions(topicId=RHKM682KQYyOfF1XsOSF1A, partitions=[0]), TopicPartitions(topicId=rKx9q1JmS1uP-ug_cj56ug, partitions=[0]), TopicPartitions(topicId=I7EtFwesTRubnj-VHClqbQ, partitions=[2]), TopicPartitions(topicId=ydAln6IUTZe-od9UUkn3rg, partitions=[2])], partitionsPendingRevocation=[]) at version 0)) from __consumer_offsets-4 at offset 3889549 with producer id -1 and producer epoch -1 failed..}}


{{java.lang.IllegalStateException: Cannot set the epoch of RHKM682KQYyOfF1XsOSF1A-0 to 854659 because the partition is still owned at epoch 853490}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)