You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "squah-confluent (via GitHub)" <gi...@apache.org> on 2025/04/16 18:28:45 UTC
[PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
squah-confluent opened a new pull request, #19495:
URL: https://github.com/apache/kafka/pull/19495
When deleting pending transactional offsets, we must preserve the list
of groups associated with the producer ID, otherwise we cannot clean up
the list of pending transactions for the group once the transaction is
committed or aborted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #19495:
URL: https://github.com/apache/kafka/pull/19495#discussion_r2047494613
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -256,7 +263,7 @@ private OffsetAndMetadata remove(
if (partitionOffsets.isEmpty())
topicOffsets.remove(topic);
- if (topicOffsets.isEmpty())
+ if (!preserveGroups && topicOffsets.isEmpty())
offsetsByGroup.remove(groupId);
Review Comment:
We use `offsetsByGroup` to clean up the list of open transactions per group.
https://github.com/apache/kafka/blob/6e26ec06bb895ff236cf1a402827d95e8d3c8c4d/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java#L1046-L1054
If we do not maintain an entry here, the clean up will not happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent closed pull request #19495: KAFKA-19164: Keep track of groups when deleting transactional offsets
URL: https://github.com/apache/kafka/pull/19495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19495:
URL: https://github.com/apache/kafka/pull/19495#discussion_r2047758771
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -256,7 +263,7 @@ private OffsetAndMetadata remove(
if (partitionOffsets.isEmpty())
topicOffsets.remove(topic);
- if (topicOffsets.isEmpty())
+ if (!preserveGroups && topicOffsets.isEmpty())
offsetsByGroup.remove(groupId);
Review Comment:
I guess we could also just remove from the openTransactionsByGroup here as well? Not a strong opinion either way. If we do it this way we should also clean it up when we remove from openTransactionByGroup? I don't think we want to preserve it forever?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on PR #19495:
URL: https://github.com/apache/kafka/pull/19495#issuecomment-2866899747
Closing this since #19497 fixes the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #19495:
URL: https://github.com/apache/kafka/pull/19495#discussion_r2047494613
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -256,7 +263,7 @@ private OffsetAndMetadata remove(
if (partitionOffsets.isEmpty())
topicOffsets.remove(topic);
- if (topicOffsets.isEmpty())
+ if (!preserveGroups && topicOffsets.isEmpty())
offsetsByGroup.remove(groupId);
Review Comment:
We use `offsetsByGroup` to clean up the list of open transactions per group.
https://github.com/apache/kafka/blob/6e26ec06bb895ff236cf1a402827d95e8d3c8c4d/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java#L1046-L1054
If we do not maintain an entry here, the clean up will not happen. Later, we will not delete the group when we expire its last offset.
https://github.com/apache/kafka/blob/6e26ec06bb895ff236cf1a402827d95e8d3c8c4d/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java#L887-L888
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19164: Keep track of groups when deleting transactional offsets [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #19495:
URL: https://github.com/apache/kafka/pull/19495#discussion_r2050629765
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/OffsetMetadataManager.java:
##########
@@ -256,7 +263,7 @@ private OffsetAndMetadata remove(
if (partitionOffsets.isEmpty())
topicOffsets.remove(topic);
- if (topicOffsets.isEmpty())
+ if (!preserveGroups && topicOffsets.isEmpty())
offsetsByGroup.remove(groupId);
Review Comment:
I guess we could. It depends on whether we are trying to track whether a group has an open transaction with offset commits that will take effect when committed or whether a group has an open transaction at all.
We certainly shouldn't delete groups with uncommitted offsets, since they could be committed afterwards and would then not get cleaned up. I'm just not sure about groups with deleted uncommitted offsets that are a no-op when committed.
@dajac What should we be doing in the second case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org