You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dongnuo123 (via GitHub)" <gi...@apache.org> on 2024/08/05 18:34:21 UTC
[PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
dongnuo123 opened a new pull request, #16800:
URL: https://github.com/apache/kafka/pull/16800
This patch enables the static member using the old protocol to be replaced in ConsumerGroupHeartbeat even if it hasn't sent a request to leave the group.
This helps in cases where existing static member rejoins and triggers a group upgrade, because the classic static consumer doesn't send leave group request before shutting down.
JIRA: https://issues.apache.org/jira/browse/KAFKA-17228
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1705097927
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
+
+ // Newly joining static member bumps the group epoch. A new target assignment is computed.
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 1),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 1),
Review Comment:
For my understanding, the group epoch is bumped in this case because the group was converted from classic to consumer, right?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
Review Comment:
Do we need this line?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
+
+ // Newly joining static member bumps the group epoch. A new target assignment is computed.
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 1),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 1),
+
+ // The newly created static member takes the assignment from the existing member.
+ // Bump its member epoch and transition to STABLE.
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedFinalConsumerMember)
+ );
+
+ assertRecordsEquals(expectedRecords, result.records());
+ context.assertSessionTimeout(groupId, newMemberId, 45000);
+
+ // Simulate a failed replay. The context is rolled back and the group is converted back to the classic group.
+ context.rollback();
+ assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
Review Comment:
I suppose that we could remove this part from this test. What do you think?
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
+
+ // Newly joining static member bumps the group epoch. A new target assignment is computed.
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 1),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 1),
+
+ // The newly created static member takes the assignment from the existing member.
+ // Bump its member epoch and transition to STABLE.
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedFinalConsumerMember)
+ );
+
+ assertRecordsEquals(expectedRecords, result.records());
+ context.assertSessionTimeout(groupId, newMemberId, 45000);
+
+ // Simulate a failed replay. The context is rolled back and the group is converted back to the classic group.
+ context.rollback();
+ assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
+ }
+
Review Comment:
Leaving this comment here but it is not related to this line at all.
I wonder if we should have a similar test but without the group conversion. We could start with a consumer group with a classic member and do the rest. I would like to ensure that this works too and that the group epoch is not bumped in this case. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1706680702
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
Review Comment:
For example:
```java
Map<String, byte[]> assignments = Collections.singletonMap(
memberId,
Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
Collections.singletonList(new TopicPartition(fooTopicName, 0))
)))
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1706667554
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
Review Comment:
nit: It would be beneficial to extract this string literal as a variable since it is used more than once and is not a short string.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
Review Comment:
nit: The **double brace initialization** is not easy to understand and actually creates an anonymous class. Since you only have a single entry, consider replacing it with `Collections.singletonMap()` for a more concise and easily understandable solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1707982335
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
+
+ // Newly joining static member bumps the group epoch. A new target assignment is computed.
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 1),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 1),
+
+ // The newly created static member takes the assignment from the existing member.
+ // Bump its member epoch and transition to STABLE.
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedFinalConsumerMember)
+ );
+
+ assertRecordsEquals(expectedRecords, result.records());
+ context.assertSessionTimeout(groupId, newMemberId, 45000);
+
+ // Simulate a failed replay. The context is rolled back and the group is converted back to the classic group.
+ context.rollback();
+ assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
+ }
+
Review Comment:
Yes it makes sense to me. Added a new test that doesn't bump the epoch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #16800:
URL: https://github.com/apache/kafka/pull/16800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1706687997
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
Review Comment:
It would be great to extract this line into a helper function.
The line is difficult to understand, and since it is encapsulated within a large `Arrays.asList` block, a helper function would be more advantageous than a distant local variable, enhancing code readability and clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on PR #16800:
URL: https://github.com/apache/kafka/pull/16800#issuecomment-2274445322
> Do we need to do anything on the downgrade path? In the classic protocol, a member rejoining with an existing static id fenced any existing one. Does it already work like this?
Yes it does, so nothing needed to change on the downgrade path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1707981191
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
Review Comment:
It's not necessary anymore if we remove testing the failed replay. Removed it.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
+
+ // Newly joining static member bumps the group epoch. A new target assignment is computed.
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 1),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 1),
Review Comment:
Yes it's correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17228: Static member using new protocol should always replace the one using the old protocol [kafka]
Posted by "dongnuo123 (via GitHub)" <gi...@apache.org>.
dongnuo123 commented on code in PR #16800:
URL: https://github.com/apache/kafka/pull/16800#discussion_r1707996818
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -10123,6 +10123,172 @@ public void testConsumerGroupHeartbeatWithPreparingRebalanceClassicGroup() throw
assertEquals(group, context.groupMetadataManager.getOrMaybeCreateClassicGroup("group-id", false));
}
+ @Test
+ public void testConsumerGroupHeartbeatToClassicGroupFromExistingStaticMember() {
+ String groupId = "group-id";
+ String memberId = "member-id";
+ String instanceId = "instance-id";
+ Uuid fooTopicId = Uuid.randomUuid();
+ String fooTopicName = "foo";
+
+ MetadataImage metadataImage = new MetadataImageBuilder()
+ .addTopic(fooTopicId, fooTopicName, 1)
+ .addRacks()
+ .build();
+
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .withConsumerGroupMigrationPolicy(ConsumerGroupMigrationPolicy.UPGRADE)
+ .withConsumerGroupAssignors(Collections.singletonList(new NoOpPartitionAssignor()))
+ .withMetadataImage(metadataImage)
+ .build();
+
+ JoinGroupRequestData.JoinGroupRequestProtocolCollection protocols = new JoinGroupRequestData.JoinGroupRequestProtocolCollection(1);
+ protocols.add(new JoinGroupRequestData.JoinGroupRequestProtocol()
+ .setName(NoOpPartitionAssignor.NAME)
+ .setMetadata(Utils.toArray(ConsumerProtocol.serializeSubscription(new ConsumerPartitionAssignor.Subscription(
+ Collections.singletonList(fooTopicName),
+ null,
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ ))))
+ );
+
+ Map<String, byte[]> assignments = new HashMap<String, byte[]>() {
+ {
+ put(
+ memberId,
+ Utils.toArray(ConsumerProtocol.serializeAssignment(new ConsumerPartitionAssignor.Assignment(
+ Collections.singletonList(new TopicPartition(fooTopicName, 0))
+ )))
+ );
+ }
+ };
+
+ // Create a stable classic group with a static member.
+ ClassicGroup group = context.createClassicGroup(groupId);
+ group.setProtocolName(Optional.ofNullable(NoOpPartitionAssignor.NAME));
+ group.add(
+ new ClassicGroupMember(
+ memberId,
+ Optional.ofNullable(instanceId),
+ "client",
+ "localhost/127.0.0.1",
+ 10000,
+ 5000,
+ "consumer",
+ protocols,
+ assignments.get(memberId)
+ )
+ );
+
+ group.transitionTo(PREPARING_REBALANCE);
+ group.transitionTo(COMPLETING_REBALANCE);
+ group.transitionTo(STABLE);
+
+ context.replay(CoordinatorRecordHelpers.newGroupMetadataRecord(group, assignments, metadataImage.features().metadataVersion()));
+ context.commit();
+ group = context.groupMetadataManager.getOrMaybeCreateClassicGroup(groupId, false);
+
+ // The static member rejoins with new protocol, triggering the upgrade.
+ CoordinatorResult<ConsumerGroupHeartbeatResponseData, CoordinatorRecord> result = context.consumerGroupHeartbeat(
+ new ConsumerGroupHeartbeatRequestData()
+ .setGroupId(groupId)
+ .setInstanceId(instanceId)
+ .setRebalanceTimeoutMs(5000)
+ .setServerAssignor(NoOpPartitionAssignor.NAME)
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setTopicPartitions(Collections.emptyList()));
+
+ ConsumerGroupMember expectedClassicMember = new ConsumerGroupMember.Builder(memberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setClientId("client")
+ .setClientHost("localhost/127.0.0.1")
+ .setSubscribedTopicNames(Collections.singletonList(fooTopicName))
+ .setRebalanceTimeoutMs(10000)
+ .setClassicMemberMetadata(
+ new ConsumerGroupMemberMetadataValue.ClassicMemberMetadata()
+ .setSessionTimeoutMs(5000)
+ .setSupportedProtocols(ConsumerGroupMember.classicProtocolListFromJoinRequestProtocolCollection(protocols))
+ )
+ .setAssignedPartitions(mkAssignment(
+ mkTopicAssignment(fooTopicId, 0)))
+ .build();
+
+ String newMemberId = result.response().memberId();
+ ConsumerGroupMember expectedReplacingConsumerMember = new ConsumerGroupMember.Builder(newMemberId)
+ .setInstanceId(instanceId)
+ .setMemberEpoch(0)
+ .setPreviousMemberEpoch(0)
+ .setState(MemberState.STABLE)
+ .setClientId(expectedClassicMember.clientId())
+ .setClientHost(expectedClassicMember.clientHost())
+ .setSubscribedTopicNames(new ArrayList<>(expectedClassicMember.subscribedTopicNames()))
+ .setRebalanceTimeoutMs(expectedClassicMember.rebalanceTimeoutMs())
+ .setAssignedPartitions(expectedClassicMember.assignedPartitions())
+ .setClassicMemberMetadata(expectedClassicMember.classicMemberMetadata().get())
+ .build();
+
+ ConsumerGroupMember expectedFinalConsumerMember = new ConsumerGroupMember.Builder(expectedReplacingConsumerMember)
+ .setMemberEpoch(1)
+ .setServerAssignorName(NoOpPartitionAssignor.NAME)
+ .setRebalanceTimeoutMs(5000)
+ .setClassicMemberMetadata(null)
+ .build();
+
+ List<CoordinatorRecord> expectedRecords = Arrays.asList(
+ // The existing classic group tombstone.
+ CoordinatorRecordHelpers.newGroupMetadataTombstoneRecord(groupId),
+
+ // Create the new consumer group with the static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedClassicMember),
+ CoordinatorRecordHelpers.newGroupEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, memberId, expectedClassicMember.assignedPartitions()),
+ CoordinatorRecordHelpers.newTargetAssignmentEpochRecord(groupId, 0),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedClassicMember),
+
+ // Remove the static member because the rejoining member replaces it.
+ CoordinatorRecordHelpers.newCurrentAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newTargetAssignmentTombstoneRecord(groupId, memberId),
+ CoordinatorRecordHelpers.newMemberSubscriptionTombstoneRecord(groupId, memberId),
+
+ // Create the new static member.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedReplacingConsumerMember),
+ CoordinatorRecordHelpers.newTargetAssignmentRecord(groupId, newMemberId, mkAssignment(mkTopicAssignment(fooTopicId, 0))),
+ CoordinatorRecordHelpers.newCurrentAssignmentRecord(groupId, expectedReplacingConsumerMember),
+
+ // The static member rejoins the new consumer group.
+ CoordinatorRecordHelpers.newMemberSubscriptionRecord(groupId, expectedFinalConsumerMember),
+
+ // The subscription metadata hasn't been updated during the conversion, so a new one is computed.
+ CoordinatorRecordHelpers.newGroupSubscriptionMetadataRecord(groupId, new HashMap<String, TopicMetadata>() {
+ {
+ put(fooTopicName, new TopicMetadata(fooTopicId, fooTopicName, 1, new HashMap<Integer, Set<String>>() {
+ {
+ put(0, new HashSet<>(Arrays.asList("rack0", "rack1")));
+ }
+ }));
+ }
+ }),
Review Comment:
Thanks for the comment! Just realized we have `mkMapOfPartitionRacks` for quick initialization of such maps Let me also replace similar cases with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org