[jira] [Assigned] (KAFKA-19862) Group coordinator loading may fail when there is concurrent compaction

["Sean Quah (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-19862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Quah reassigned KAFKA-19862:
---------------------------------

    Assignee: Izzy Harker  (was: Sean Quah)

> Group coordinator loading may fail when there is concurrent compaction
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-19862
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19862
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator
>            Reporter: Sean Quah
>            Assignee: Izzy Harker
>            Priority: Blocker
>             Fix For: 4.2.0
>
>
> For consumer and streams groups, we reject replay of {{Consumer/StreamsGroupCurrentMemberAssignment}} records when we detect a partition / task is already owned by another member.
> During group coordinator load, we replay the records in {{{}__consumer_offsets{}}}. When compaction is running concurrently, we can load uncompacted data, followed by a newly swapped in compacted segment, followed by the uncompacted head of the log. This allows for situations where the record unassigning a partition/task is missed during loading.
> eg.
> We can load a record \{ Member A is assigned partition X },
> then miss the record \{ Member A is unassigned partition X },
> then load the record \{ Member B is assigned partition X }, which fails with an exception like
> {{[GroupCoordinator id=2] Failed to load metadata from __consumer_offsets-4 with epoch 10 due to java.lang.RuntimeException: Replaying record CoordinatorRecord(key=ConsumerGroupCurrentMemberAssignmentKey(groupId='...', memberId='ZxHk7W53S_aHFdpxYc-_Jw'), value=ApiMessageAndVersion(ConsumerGroupCurrentMemberAssignmentValue(memberEpoch=854659, previousMemberEpoch=854633, state=0, assignedPartitions=[TopicPartitions(topicId=9lL1aTMuSC22QAXsHgzhew, partitions=[1, 2]), TopicPartitions(topicId=RHKM682KQYyOfF1XsOSF1A, partitions=[0]), TopicPartitions(topicId=rKx9q1JmS1uP-ug_cj56ug, partitions=[0]), TopicPartitions(topicId=I7EtFwesTRubnj-VHClqbQ, partitions=[2]), TopicPartitions(topicId=ydAln6IUTZe-od9UUkn3rg, partitions=[2])], partitionsPendingRevocation=[]) at version 0)) from __consumer_offsets-4 at offset 3889549 with producer id -1 and producer epoch -1 failed..}}
> {{java.lang.IllegalStateException: Cannot set the epoch of RHKM682KQYyOfF1XsOSF1A-0 to 854659 because the partition is still owned at epoch 853490}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)