You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dongnuo123 (via GitHub)" <gi...@apache.org> on 2024/02/28 22:46:11 UTC
[PR] KAFKA-16313: offline group protocol upgrade [kafka]
dongnuo123 opened a new pull request, #15442:
URL: https://github.com/apache/kafka/pull/15442
This patch enables the empty classic groups to upgrade to classic groups.
For the common groups, the upgrade is triggered if the classic group is empty.
For the existing simple groups, the upgrade is triggered if they try to commit offsets. The newly created simple groups will be upgraded to consumer groups immediately.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1516473475
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
Yeah, the replay will fail if you add the check. Need to think about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1507837970
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -474,6 +474,8 @@ public CoordinatorResult<OffsetCommitResponseData, Record> commitOffset(
final List<Record> records = new ArrayList<>();
final long currentTimeMs = time.milliseconds();
final OptionalLong expireTimestampMs = expireTimestampMs(request.retentionTimeMs(), currentTimeMs);
+ groupMetadataManager.maybeUpgradeEmptyGroup(group.groupId(), records, true);
+ final int initialRecordsSize = records.size();
Review Comment:
My question was more about the `maybeUpgradeEmptyGroup` called here. Why do we need it? This seems to be orthogonal to the offset metadata manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1508759431
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3501,42 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade/Downgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeMigrateEmptyGroup(String groupId, List<Record> records, boolean isUpgrade) {
+ if ((isUpgrade && validateOfflineUpgrade(groupId)) ||
+ (!isUpgrade && validateOfflineDowngrade(groupId))) {
+ deleteGroup(groupId, records);
+ removeGroup(groupId);
Review Comment:
Do we actually replay records in this particular case? The replay handling for the classic protocol is a big different from all the other ones. See here: https://github.com/apache/kafka/pull/15442/files#diff-00f0f81cf13e66781777d94f7d2e68a581663385c37e98792507f2294c91bb09R1978. When the coordinator result contains a `appendFuture`, the write operation skips the replay if I remember correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1517690579
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
I think that the weirdness comes from the fact that `getOrMaybeCreateConsumerGroup` actually put the new groups into the map. Ideally, the entire consumerGroupHeartbeat method should not alter the timeline data structure and only rely on the replay to update it. I wonder if we should fix this. If we do, they the replay in this case would be just fine. If we do this, it also means that we don't have to remove the classic group from the map too. We only need to generate the tombstone.
We don't have this issue on the classic group case because we don't replay records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1516453885
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
Got it. Let me add a type check when replaying classic group tombstone. Now it will be like (1) remove the classic group from the map, (2) create the consumer group, (3) the classic group tombstone replay fails because the group is no longer a classic group and (4) replay the consumer group records, which only updates the group epoch because the consumer group already exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1507302228
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -474,6 +474,8 @@ public CoordinatorResult<OffsetCommitResponseData, Record> commitOffset(
final List<Record> records = new ArrayList<>();
final long currentTimeMs = time.milliseconds();
final OptionalLong expireTimestampMs = expireTimestampMs(request.retentionTimeMs(), currentTimeMs);
+ groupMetadataManager.maybeUpgradeEmptyGroup(group.groupId(), records, true);
+ final int initialRecordsSize = records.size();
Review Comment:
Why are we doing this here? I am not sure to follow.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,59 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+
+ if (group == null || group.type() == CONSUMER) {
+ return false;
+ }
+
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ if (!classicGroup.isEmpty()) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ /**
+ * Upgrade the empty classic group to a consumer group if it's valid.
+ *
+ * @param groupId The group id to be updated.
+ * @param records The list of records to delete the classic group and create the consumer group.
+ * @param isSimpleGroup The boolean indicating whether the group to be updated is a simple group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records, boolean isSimpleGroup) {
+ if (validateOfflineUpgrade(groupId)) {
+ final long currentTimeMs = time.milliseconds();
+ ClassicGroup classicGroup = getOrMaybeCreateClassicGroup(groupId, false);
+ int groupEpoch = classicGroup.generationId();
+
+ // Replace the classic group with a new consumer group.
+ ConsumerGroup consumerGroup = getOrMaybeCreateConsumerGroup(groupId, true);
+ // We don't create the tombstone because the replay will remove the newly created consumer group.
+ removeGroup(groupId);
+ groups.put(groupId, consumerGroup);
+ metrics.onConsumerGroupStateTransition(null, consumerGroup.state());
+
+ if (!isSimpleGroup) {
+ records.add(newGroupSubscriptionMetadataRecord(
+ groupId,
+ consumerGroup.computeSubscriptionMetadata(classicGroup.subscribedTopics(), metadataImage.topics(), metadataImage.cluster())
+ ));
Review Comment:
This does not seem necessary. If the group is empty, the subscribedTopics will also be empty. As I said earlier, we can let the handling of the new member create the new subscription.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1508436563
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3501,42 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade/Downgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeMigrateEmptyGroup(String groupId, List<Record> records, boolean isUpgrade) {
+ if ((isUpgrade && validateOfflineUpgrade(groupId)) ||
+ (!isUpgrade && validateOfflineDowngrade(groupId))) {
+ deleteGroup(groupId, records);
+ removeGroup(groupId);
Review Comment:
There is a bug for downgrade. The sequence of downgrading now is 1) consumer group gets deleted 2) classicGroupJoin creates a classic group and put it into `groupMetadataManager.groups`3) replaying consumer group tombstone from `deleteGroup` where `groups.get(groupId)` should be a consumer group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1515850376
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,56 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records) {
+ if (validateOfflineUpgrade(groupId)) {
+ deleteGroup(groupId, records);
+ removeGroup(groupId);
Review Comment:
nit: Those two methods next to each others look a bit weird. I wonder if we should rename `deleteGroup` to `createGroupTombstoneRecords`.
Another thing is that we actually get the group from the map in each method called here. I wonder if we should inline and simplify it. For instance, we could lookup the group as the first thing in this method, check the condition, then call `createGroupTombstoneRecords` on the group and finally remove the group from the map. What do you think?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,56 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records) {
+ if (validateOfflineUpgrade(groupId)) {
Review Comment:
nit: I wonder if naming it `isEmptyClassicGroup` would be better.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,56 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records) {
Review Comment:
nit: Could we keep it private? It does not seems necessary to expose it. I also wonder if we should name it `maybeDeleteEmptyClassicGroup`. What do you think?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList()));
+ assertEquals(0, result.response().errorCode());
+ assertEquals(Group.GroupType.CONSUMER,
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup(classicGroupId, false).type());
Review Comment:
Should we verify the records generated here?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
Review Comment:
This feels a bit weird in this test as it does not align with its name. How about putting it in a separate test?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList()));
+ assertEquals(0, result.response().errorCode());
+ assertEquals(Group.GroupType.CONSUMER,
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup(classicGroupId, false).type());
+ }
+
+ @Test
+ public void testClassicGroupJoinWithEmptyConsumerGroup() throws Exception {
+ String consumerGroupId = "consumer-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(memberId);
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
Review Comment:
Similarly, I think that we should put the non empty case in a separate test case.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList()));
+ assertEquals(0, result.response().errorCode());
+ assertEquals(Group.GroupType.CONSUMER,
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup(classicGroupId, false).type());
+ }
+
+ @Test
+ public void testClassicGroupJoinWithEmptyConsumerGroup() throws Exception {
+ String consumerGroupId = "consumer-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(memberId);
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
Review Comment:
An empty group should have the following records:
* GroupEpoch
* GroupSubscriptionMetadata (empty)
* TargetAssignmentEpoch
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,56 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records) {
+ if (validateOfflineUpgrade(groupId)) {
+ deleteGroup(groupId, records);
+ removeGroup(groupId);
+ }
+ }
+
+ /**
+ * Downgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeDowngradeEmptyGroup(String groupId, List<Record> records) {
Review Comment:
The same comments apply to this one.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
Talking about records, I was actually wondering how their replay could work in this case. My understanding is that (1) we remove the classic group from the map directly, (2) then create the consumer group, (3) then replay the classic group deletion tombstone and (4) finally replay the consumer group records. I was expecting (3) to fail because the classic group is not there anymore. It actually works because the deletion does not check the group type in this case. I wonder if we should strengthen this path. For consumer group, the replay of tombstones only work if the group type is correct.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
Review Comment:
`testMaybeUpgradeEmptyGroup` and `testMaybeDowngradeEmptyGroup` seem a bit redundant now that we have the others. Do they still bring something?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList()));
+ assertEquals(0, result.response().errorCode());
+ assertEquals(Group.GroupType.CONSUMER,
+ context.groupMetadataManager.getOrMaybeCreateConsumerGroup(classicGroupId, false).type());
+ }
+
+ @Test
+ public void testClassicGroupJoinWithEmptyConsumerGroup() throws Exception {
+ String consumerGroupId = "consumer-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(memberId);
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+
+ JoinGroupRequestData request = new GroupMetadataManagerTestContext.JoinGroupRequestBuilder()
+ .withGroupId(consumerGroupId)
+ .withMemberId(UNKNOWN_MEMBER_ID)
+ .withDefaultProtocolTypeAndProtocols()
+ .build();
+
+ GroupMetadataManagerTestContext.JoinResult joinResult1 = context.sendClassicGroupJoin(request);
+ assertEquals(Errors.GROUP_ID_NOT_FOUND.code(), joinResult1.joinFuture.get().errorCode());
+
+ // Remove the member. The consumer group becomes empty.
+ context.replay(RecordHelpers.newCurrentAssignmentTombstoneRecord(consumerGroupId, memberId));
+ context.replay(RecordHelpers.newMemberSubscriptionTombstoneRecord(consumerGroupId, memberId));
+
+ GroupMetadataManagerTestContext.JoinResult joinResult2 = context.sendClassicGroupJoin(request, true);
+ assertNotEquals(Errors.GROUP_ID_NOT_FOUND.code(), joinResult2.joinFuture.get().errorCode());
+ assertEquals(Group.GroupType.CLASSIC,
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(consumerGroupId, false).type());
Review Comment:
Should we assert the records here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1516858539
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
Maybe we can remove the group only if the group is classic when replaying group metadata tombstone. It's a little strange but I can't think of another way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1507823457
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -474,6 +474,8 @@ public CoordinatorResult<OffsetCommitResponseData, Record> commitOffset(
final List<Record> records = new ArrayList<>();
final long currentTimeMs = time.milliseconds();
final OptionalLong expireTimestampMs = expireTimestampMs(request.retentionTimeMs(), currentTimeMs);
+ groupMetadataManager.maybeUpgradeEmptyGroup(group.groupId(), records, true);
+ final int initialRecordsSize = records.size();
Review Comment:
This is because there is a `if (!records.isEmpty())` in L513. If any new records are added we need ` if (records.size() > initialRecordsSize)` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1507296782
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,59 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+
+ if (group == null || group.type() == CONSUMER) {
+ return false;
+ }
+
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ if (!classicGroup.isEmpty()) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ /**
+ * Upgrade the empty classic group to a consumer group if it's valid.
+ *
+ * @param groupId The group id to be updated.
+ * @param records The list of records to delete the classic group and create the consumer group.
+ * @param isSimpleGroup The boolean indicating whether the group to be updated is a simple group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records, boolean isSimpleGroup) {
Review Comment:
In this particular case where the group is empty and a new member joins, my understanding is that we only need to delete the previous group. The new group will be automatically created with the new member. Am I missing something?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,59 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+
+ if (group == null || group.type() == CONSUMER) {
+ return false;
+ }
+
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ if (!classicGroup.isEmpty()) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ /**
+ * Upgrade the empty classic group to a consumer group if it's valid.
+ *
+ * @param groupId The group id to be updated.
+ * @param records The list of records to delete the classic group and create the consumer group.
+ * @param isSimpleGroup The boolean indicating whether the group to be updated is a simple group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records, boolean isSimpleGroup) {
+ if (validateOfflineUpgrade(groupId)) {
+ final long currentTimeMs = time.milliseconds();
+ ClassicGroup classicGroup = getOrMaybeCreateClassicGroup(groupId, false);
+ int groupEpoch = classicGroup.generationId();
+
+ // Replace the classic group with a new consumer group.
+ ConsumerGroup consumerGroup = getOrMaybeCreateConsumerGroup(groupId, true);
+ // We don't create the tombstone because the replay will remove the newly created consumer group.
Review Comment:
I think that we must write a tombstone for the old group. As you said, the map will be updated based on the records with the new group. However, we also need to compact the old record for the group and the only way to do it is to write a tombstone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1508759431
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3501,42 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CLASSIC && group.isEmpty();
+ }
+
+ /**
+ * A group can be downgraded offline if it's a consumer group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline downgrade is valid.
+ */
+ private boolean validateOfflineDowngrade(String groupId) {
+ Group group = groups.get(groupId);
+ return group != null && group.type() == CONSUMER && group.isEmpty();
+ }
+
+ /**
+ * Upgrade/Downgrade the empty group if it's valid.
+ *
+ * @param groupId The group id to be migrated.
+ * @param records The list of records to delete the previous group.
+ */
+ public void maybeMigrateEmptyGroup(String groupId, List<Record> records, boolean isUpgrade) {
+ if ((isUpgrade && validateOfflineUpgrade(groupId)) ||
+ (!isUpgrade && validateOfflineDowngrade(groupId))) {
+ deleteGroup(groupId, records);
+ removeGroup(groupId);
Review Comment:
Do we actually replay records in this particular case? The replay handling for the classic protocol is a big different from all the other ones. See here: https://github.com/apache/kafka/pull/15442/files#diff-00f0f81cf13e66781777d94f7d2e68a581663385c37e98792507f2294c91bb09R1978.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 closed pull request #15442: KAFKA-16313: offline group protocol migration
URL: https://github.com/apache/kafka/pull/15442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1506768674
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1015,6 +1017,7 @@ private CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> consumerGr
// Get or create the consumer group.
boolean createIfNotExists = memberEpoch == 0;
+ maybeUpgradeEmptyGroup(groupId, records, false);
Review Comment:
I think we can add check for migration config here which doesn't affect simple group upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol upgrade [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1507826893
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -3500,6 +3503,59 @@ public void maybeDeleteGroup(String groupId, List<Record> records) {
}
}
+ /**
+ * A group can be upgraded offline if it's a classic group and empty.
+ *
+ * @param groupId The group to be validated.
+ * @return true if the offline upgrade is valid.
+ */
+ private boolean validateOfflineUpgrade(String groupId) {
+ Group group = groups.get(groupId);
+
+ if (group == null || group.type() == CONSUMER) {
+ return false;
+ }
+
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ if (!classicGroup.isEmpty()) {
+ return false;
+ } else {
+ return true;
+ }
+ }
+
+ /**
+ * Upgrade the empty classic group to a consumer group if it's valid.
+ *
+ * @param groupId The group id to be updated.
+ * @param records The list of records to delete the classic group and create the consumer group.
+ * @param isSimpleGroup The boolean indicating whether the group to be updated is a simple group.
+ */
+ public void maybeUpgradeEmptyGroup(String groupId, List<Record> records, boolean isSimpleGroup) {
Review Comment:
I agree with it in the case of upgrading if a new member joining the empty group, but for the simple group we still need to create the new group. Is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1516784744
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
Maybe we can remove the group only if the group is classic when replaying GroupMetadataTombstone. It's also a bit strange but I can't think of another way..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16313: offline group protocol migration [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #15442:
URL: https://github.com/apache/kafka/pull/15442#discussion_r1516784744
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9607,151 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testMaybeUpgradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ // A consumer group can't be upgraded.
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A non-empty classic group can't be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty classic group can be upgraded.
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ context.groupMetadataManager.maybeUpgradeEmptyGroup(classicGroupId, records);
+ assertEquals(Arrays.asList(RecordHelpers.newGroupMetadataTombstoneRecord(classicGroupId)), records);
+ }
+
+ @Test
+ public void testMaybeDowngradeEmptyGroup() {
+ String classicGroupId = "classic-group-id";
+ String consumerGroupId = "consumer-group-id";
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+ context.replay(RecordHelpers.newGroupEpochRecord(consumerGroupId, 10));
+
+ List<Record> records = new ArrayList<>();
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // A classic group can't be downgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+
+ // An empty consumer group can be upgraded.
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(consumerGroupId, records);
+ assertEquals(Arrays.asList(
+ RecordHelpers.newTargetAssignmentEpochTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupSubscriptionMetadataTombstoneRecord(consumerGroupId),
+ RecordHelpers.newGroupEpochTombstoneRecord(consumerGroupId)), records);
+ records.clear();
+
+ // A non-empty consumer group can't be downgraded.
+ ConsumerGroupMember.Builder memberBuilder = new ConsumerGroupMember.Builder(Uuid.randomUuid().toString());
+ context.replay(RecordHelpers.newMemberSubscriptionRecord(consumerGroupId, memberBuilder.build()));
+ context.groupMetadataManager.maybeDowngradeEmptyGroup(classicGroupId, records);
+ assertEquals(Collections.emptyList(), records);
+ }
+
+ @Test
+ public void testConsumerGroupHeartbeatWithEmptyClassicGroup() {
+ String classicGroupId = "classic-group-id";
+ String memberId = Uuid.randomUuid().toString();
+ MockPartitionAssignor assignor = new MockPartitionAssignor("range");
+ assignor.prepareGroupAssignment(new GroupAssignment(Collections.emptyMap()));
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withAssignors(Collections.singletonList(assignor))
+ .build();
+ ClassicGroup classicGroup = new ClassicGroup(
+ new LogContext(),
+ classicGroupId,
+ EMPTY,
+ context.time,
+ context.metrics
+ );
+ context.replay(RecordHelpers.newGroupMetadataRecord(classicGroup, classicGroup.groupAssignment(), MetadataVersion.latestTesting()));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(PREPARING_REBALANCE);
+ assertThrows(GroupIdNotFoundException.class, () ->
+ context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(classicGroupId)
+ .setMemberId(memberId)
+ .setMemberEpoch(0)
+ .setServerAssignor("range")
+ .setRebalanceTimeoutMs(5000)
+ .setSubscribedTopicNames(Arrays.asList("foo", "bar"))
+ .setTopicPartitions(Collections.emptyList())));
+
+ context.groupMetadataManager.getOrMaybeCreateClassicGroup(classicGroupId, false).transitionTo(EMPTY);
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, Record> result = context.consumerGroupHeartbeat(
Review Comment:
Maybe we can remove the group only if the group is classic when replaying GroupMetadataTombstone. It's also a bit strange but I can't think of another way..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org