You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "jeffkbkim (via GitHub)" <gi...@apache.org> on 2024/02/29 17:34:12 UTC
[PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
jeffkbkim opened a new pull request, #15446:
URL: https://github.com/apache/kafka/pull/15446
In the new coordinator, we have lingering timers (heartbeats/revocation timeouts/join groups/sync groups). For classic groups, we also have awaiting join/sync futures that are never completed. This patch cancels all existing timers and completes all awaiting futures when a group is unloaded.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1509452557
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,63 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+
+ consumerGroup.members().values().forEach(member -> {
+ timer.cancel(consumerGroupSessionTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ timer.cancel(consumerGroupRevocationTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ });
+
+ break;
+ case CLASSIC:
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for generation {}.",
+ classicGroup.groupId(), classicGroup.generationId());
+
+ classicGroup.transitionTo(DEAD);
+ switch (classicGroup.previousState()) {
+ case EMPTY:
+ case DEAD:
+ break;
+ case PREPARING_REBALANCE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeJoinFuture(member, new JoinGroupResponseData()
+ .setMemberId(member.memberId())
+ .setMembers(Collections.emptyList())
+ .setGenerationId(NO_GENERATION)
Review Comment:
will remove them. sorry for the confusion, we don't need https://issues.apache.org/jira/browse/KAFKA-16299. i don't know where i saw the default values but i remember they were set as 0 and null..
removed all NO_GENERATION and NO_PROTOCOL_NAME usages as they are the defaults
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1515878176
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##########
@@ -9607,6 +9609,104 @@ public void testOnConsumerGroupStateTransitionOnLoading() {
verify(context.metrics, times(1)).onConsumerGroupStateTransition(ConsumerGroup.ConsumerGroupState.EMPTY, null);
}
+ @Test
+ public void testClassicGroupOnUnloadedEmptyAndPreparingRebalance() throws Exception {
+ GroupMetadataManagerTestContext context = new GroupMetadataManagerTestContext.Builder()
+ .build();
+
+ ClassicGroup emptyGroup = context.createClassicGroup("empty-group");
+ assertTrue(emptyGroup.isInState(EMPTY));
+
+ ClassicGroup preparingGroup = context.createClassicGroup("preparing-group");
+ JoinGroupRequestData request = new GroupMetadataManagerTestContext.JoinGroupRequestBuilder()
Review Comment:
Is it worth having two members in the group?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -639,6 +639,7 @@ public void onLoaded(MetadataImage newImage) {
public void onUnloaded() {
timer.cancel(GROUP_EXPIRATION_KEY);
coordinatorMetrics.deactivateMetricsShard(metricsShard);
+ groupMetadataManager.onUnloaded();
Review Comment:
Should we extend unit test to cover this change?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1831,54 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloaded group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+ break;
+ case CLASSIC:
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for generation {}.",
+ classicGroup.groupId(), classicGroup.generationId());
+
+ classicGroup.transitionTo(DEAD);
+ switch (classicGroup.previousState()) {
+ case EMPTY:
+ case DEAD:
+ break;
+ case PREPARING_REBALANCE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeJoinFuture(member, new JoinGroupResponseData()
+ .setMemberId(member.memberId())
+ .setErrorCode(NOT_COORDINATOR.code()));
+
+ timer.cancel(classicGroupHeartbeatKey(classicGroup.groupId(), member.memberId()));
Review Comment:
We can also remove the cancellations for the classic group case.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1831,54 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
Review Comment:
nit: We could remove `Cancel all existing timers for the group.`
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,64 @@ public void onLoaded() {
});
}
+ // TODO: this part is done asynchronously in the old coordinator via a KafkaScheduler#schedule
+ /**
Review Comment:
That's a good point. We need to observe how it works. If it causes performance issues, we could move this to the loader and use the loader to load and unload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on PR #15446:
URL: https://github.com/apache/kafka/pull/15446#issuecomment-2030301885
Fixed. Other test failures are unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1507948503
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,64 @@ public void onLoaded() {
});
}
+ // TODO: this part is done asynchronously in the old coordinator via a KafkaScheduler#schedule
+ /**
Review Comment:
would like to point out that the existing coordinator does this asynchronously
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on PR #15446:
URL: https://github.com/apache/kafka/pull/15446#issuecomment-2018694290
thanks @dajac. I have addressed your comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #15446:
URL: https://github.com/apache/kafka/pull/15446#issuecomment-2022239076
@jeffkbkim There are failed tests that look related. Could you check them please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1515870996
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,63 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+
+ consumerGroup.members().values().forEach(member -> {
+ timer.cancel(consumerGroupSessionTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ timer.cancel(consumerGroupRevocationTimeoutKey(consumerGroup.groupId(), member.memberId()));
Review Comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1539240498
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1920,6 +1921,49 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded.
+ * ClassicGroup: Complete all awaiting join and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloaded group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+ break;
+ case CLASSIC:
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for generation {}.",
+ classicGroup.groupId(), classicGroup.generationId());
+
+ classicGroup.transitionTo(DEAD);
+ switch (classicGroup.previousState()) {
+ case EMPTY:
+ case DEAD:
+ break;
+ case PREPARING_REBALANCE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeJoinFuture(member, new JoinGroupResponseData()
+ .setMemberId(member.memberId())
+ .setErrorCode(NOT_COORDINATOR.code()));
+ });
+
+ break;
+ case COMPLETING_REBALANCE:
+ case STABLE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeSyncFuture(member, new SyncGroupResponseData()
+ .setErrorCode(NOT_COORDINATOR.code()));
+ });
+
+ removeSyncExpiration(classicGroup);
Review Comment:
I wonder if we could also remove this one for the same reason that we remove the other ones. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1539809862
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1920,6 +1921,49 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded.
+ * ClassicGroup: Complete all awaiting join and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloaded group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+ break;
+ case CLASSIC:
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for generation {}.",
+ classicGroup.groupId(), classicGroup.generationId());
+
+ classicGroup.transitionTo(DEAD);
+ switch (classicGroup.previousState()) {
+ case EMPTY:
+ case DEAD:
+ break;
+ case PREPARING_REBALANCE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeJoinFuture(member, new JoinGroupResponseData()
+ .setMemberId(member.memberId())
+ .setErrorCode(NOT_COORDINATOR.code()));
+ });
+
+ break;
+ case COMPLETING_REBALANCE:
+ case STABLE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeSyncFuture(member, new SyncGroupResponseData()
+ .setErrorCode(NOT_COORDINATOR.code()));
+ });
+
+ removeSyncExpiration(classicGroup);
Review Comment:
i missed that. thanks for the catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1508764392
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,63 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+
+ consumerGroup.members().values().forEach(member -> {
+ timer.cancel(consumerGroupSessionTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ timer.cancel(consumerGroupRevocationTimeoutKey(consumerGroup.groupId(), member.memberId()));
Review Comment:
Cancelling the timers is not necessary here because all the timers of the shard are already cancelled when we reach this point. Check the unload method in the coordinator context.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,63 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+
+ consumerGroup.members().values().forEach(member -> {
+ timer.cancel(consumerGroupSessionTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ timer.cancel(consumerGroupRevocationTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ });
+
+ break;
+ case CLASSIC:
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for generation {}.",
+ classicGroup.groupId(), classicGroup.generationId());
+
+ classicGroup.transitionTo(DEAD);
+ switch (classicGroup.previousState()) {
+ case EMPTY:
+ case DEAD:
+ break;
+ case PREPARING_REBALANCE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeJoinFuture(member, new JoinGroupResponseData()
+ .setMemberId(member.memberId())
+ .setMembers(Collections.emptyList())
+ .setGenerationId(NO_GENERATION)
+ .setErrorCode(NOT_COORDINATOR.code()));
+
+ timer.cancel(classicGroupHeartbeatKey(classicGroup.groupId(), member.memberId()));
+ });
+
+ timer.cancel(classicGroupJoinKey(classicGroup.groupId()));
+ break;
+ case COMPLETING_REBALANCE:
+ case STABLE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeSyncFuture(member, new SyncGroupResponseData()
+ .setAssignment(EMPTY_ASSIGNMENT)
Review Comment:
The assignment is empty by default too.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,63 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+
+ consumerGroup.members().values().forEach(member -> {
+ timer.cancel(consumerGroupSessionTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ timer.cancel(consumerGroupRevocationTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ });
+
+ break;
+ case CLASSIC:
+ ClassicGroup classicGroup = (ClassicGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for generation {}.",
+ classicGroup.groupId(), classicGroup.generationId());
+
+ classicGroup.transitionTo(DEAD);
+ switch (classicGroup.previousState()) {
+ case EMPTY:
+ case DEAD:
+ break;
+ case PREPARING_REBALANCE:
+ classicGroup.allMembers().forEach(member -> {
+ classicGroup.completeJoinFuture(member, new JoinGroupResponseData()
+ .setMemberId(member.memberId())
+ .setMembers(Collections.emptyList())
+ .setGenerationId(NO_GENERATION)
Review Comment:
Members are empty by default and the generation id is -1 by default too. I wonder if we could just remove those two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on code in PR #15446:
URL: https://github.com/apache/kafka/pull/15446#discussion_r1509457561
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##########
@@ -1830,6 +1832,63 @@ public void onLoaded() {
});
}
+ /**
+ * Called when the partition is unloaded. Cancel all existing timers for the group.
+ * ClassicGroup: Complete all awaiting join futures and sync futures. Transition group to Dead.
+ */
+ public void onUnloaded() {
+ groups.values().forEach(group -> {
+ switch (group.type()) {
+ case CONSUMER:
+ ConsumerGroup consumerGroup = (ConsumerGroup) group;
+ log.info("[GroupId={}] Unloading group metadata for group epoch {}.",
+ consumerGroup.groupId(), consumerGroup.groupEpoch());
+
+ consumerGroup.members().values().forEach(member -> {
+ timer.cancel(consumerGroupSessionTimeoutKey(consumerGroup.groupId(), member.memberId()));
+ timer.cancel(consumerGroupRevocationTimeoutKey(consumerGroup.groupId(), member.memberId()));
Review Comment:
then we should expect classic group timers to also be cancelled already right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16148: Implement GroupMetadataManager#onUnloaded [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #15446:
URL: https://github.com/apache/kafka/pull/15446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org