You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2026/06/25 15:17:28 UTC
[PR] MINOR: Follow-up KAFKA-20696 [kafka]
frankvicky opened a new pull request, #22676:
URL: https://github.com/apache/kafka/pull/22676
Deal the comments of #22622
- **`OffsetMetadataManager.allOffsetsExpired` → `groupHasNoOffsets`**. Per-topic / per-partition expiration check was O(topics × partitions) per group. Since the regular offset-expiration cycle already tombstones expired offsets, this stage only needs the post-state — a single null-check on `offsetsByGroup` plus an `openTransactions.contains` (O(1)). Three cycles compose cleanly: offset expiration → topology cleanup → group tombstone. The now-unused snapshot overload `OpenTransactions.contains(groupId, topic, partition, committedOffset)` is removed.
- **Per-shard batching of the conditional clear write**. All groups in one partition's eligibility read hash to the same `__consumer_offsets` partition, so the cycle now issues one `scheduleWriteOperation` per shard carrying every conditional clear for that shard — not one write per group. Added `GroupCoordinatorShard.clearStoredDescriptionTopologyEpochBatch(Map<String, Integer>)` that folds per-group GMM results into one `CoordinatorResult`.
- **Restored inline comment** on the pending-transactional-offset guard in `cleanupExpiredOffsets` — accidentally dropped during the helper-extract / inline round-trip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky merged PR #22676:
URL: https://github.com/apache/kafka/pull/22676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3490172839
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupCoordinatorServiceTopologyDescriptionTest.java:
##########
@@ -951,6 +952,32 @@ public void testCleanupCycleClearsStoredEpochOnPluginSuccess() {
verify(runtime, times(1)).scheduleWriteOperation(eq("clear-stored-topology-epoch"), eq(GROUP_TP), any());
}
+ @Test
+ public void testCleanupCycleBatchesClearWritesPerPartition() {
+ // Two eligible groups land on the same partition's eligibility read — they must trigger
+ // exactly one scheduleWriteOperation carrying both conditional clears (dajac's per-shard
Review Comment:
Can we drop the reviewer attribution here? The comment should explain what the test checks, not where the idea came from.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -1014,7 +1015,7 @@ public Map<String, Integer> listStreamsGroupsNeedingTopologyCleanup(long committ
if (!streamsGroup.isEmpty(committedOffset)) continue;
int storedEpoch = streamsGroup.storedDescriptionTopologyEpoch(committedOffset);
Review Comment:
This PR still has `deferStreamsGroupTombstoneForPluginCleanup` as a static method in the shard — @dajac asked to put this per-group logic on `shouldExpire` instead.
Suggestion: add `config` as a parameter — `shouldExpire(GroupCoordinatorConfig config)` — so `StreamsGroup` overrides it to return `false` when the plugin is configured and `storedDescriptionTopologyEpoch != -1`. Other group types ignore the parameter and keep their current behaviour. That removes `deferStreamsGroupTombstoneForPluginCleanup` entirely and the call site in `cleanupGroupMetadata` becomes a plain `if (!group.shouldExpire(config)) return;`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3490799408
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -1037,6 +1038,23 @@ public CoordinatorResult<Void, CoordinatorRecord> clearStoredDescriptionTopology
return groupMetadataManager.clearStoredDescriptionTopologyEpoch(groupId, expectedStoredEpoch);
}
+ /**
+ * Batched form of {@link #clearStoredDescriptionTopologyEpoch}: emits one conditional clear
+ * record per entry in {@code expectedStoredEpochByGroupId} in a single
+ * {@link CoordinatorResult}, so the cycle issues one {@code scheduleWriteOperation} per
+ * shard instead of one per group. Groups whose stored epoch no longer matches yield no
+ * record (the GMM-level method returns an empty list for those), matching the single-group
+ * variant's silent skip.
+ */
+ public CoordinatorResult<Void, CoordinatorRecord> clearStoredDescriptionTopologyEpochBatch(
+ Map<String, Integer> expectedStoredEpochByGroupId
+ ) {
+ List<CoordinatorRecord> records = new ArrayList<>(expectedStoredEpochByGroupId.size());
+ expectedStoredEpochByGroupId.forEach((groupId, expectedStoredEpoch) ->
+ records.addAll(groupMetadataManager.clearStoredDescriptionTopologyEpoch(groupId, expectedStoredEpoch).records()));
+ return new CoordinatorResult<>(records);
Review Comment:
`clearStoredDescriptionTopologyEpochBatch` returns `new CoordinatorResult<>(records)` which defaults to `isAtomic=true`. Because this batched write can include many independent per-group records, keeping it atomic increases the risk of `RecordBatchTooLargeException` (atomic writes cannot be split across batches) and is inconsistent with other bulk-maintenance writes in this shard that explicitly use `isAtomic=false` (e.g. `cleanupGroupMetadata` at GroupCoordinatorShard.java:1148 and `onTopicsDeleted` at :1201). Consider returning a non-atomic result here so the runtime can split the records across multiple batches if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on PR #22676:
URL: https://github.com/apache/kafka/pull/22676#issuecomment-4808062827
Deal withs the new comments from https://github.com/apache/kafka/pull/22622#discussion_r3466905977 after merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3491688658
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -1014,7 +1015,7 @@ public Map<String, Integer> listStreamsGroupsNeedingTopologyCleanup(long committ
if (!streamsGroup.isEmpty(committedOffset)) continue;
int storedEpoch = streamsGroup.storedDescriptionTopologyEpoch(committedOffset);
Review Comment:
Ahh, I think number 2 would be fine. We don't care about offsets in share groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3491704677
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -1014,7 +1015,7 @@ public Map<String, Integer> listStreamsGroupsNeedingTopologyCleanup(long committ
if (!streamsGroup.isEmpty(committedOffset)) continue;
int storedEpoch = streamsGroup.storedDescriptionTopologyEpoch(committedOffset);
Review Comment:
We should clearly express in the javadoc that this will not prevent expiration of offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #22676:
URL: https://github.com/apache/kafka/pull/22676#issuecomment-4842037498
@frankvicky I have posted two nits and this needs to be rebased. Could you do this so we can merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3497728293
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupCoordinatorShardTest.java:
##########
@@ -1808,6 +1816,10 @@ public void testCleanupGroupMetadataForShareGroup() {
when(groupMetadataManager.groupIds()).thenReturn(Set.of("group-id"));
when(groupMetadataManager.group("group-id")).thenReturn(group);
+ // Offset cleanup runs uniformly; for share groups it has nothing to do and returns
Review Comment:
nit: The real cleanupExpiredOffsets actually returns true for a share group, not false. I think we should test the real scenario here.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupCoordinatorShardTest.java:
##########
@@ -1736,29 +1737,37 @@ public void testCleanupGroupMetadataTombstoneStreamsGroupWithoutStoredTopologyEp
mock(CoordinatorMetricsShard.class)
);
- StreamsGroup streamsGroup = mock(StreamsGroup.class);
- when(streamsGroup.shouldExpire()).thenReturn(true);
- when(streamsGroup.type()).thenReturn(Group.GroupType.STREAMS);
- when(streamsGroup.storedDescriptionTopologyEpoch()).thenReturn(-1);
+ CoordinatorRecord offsetCommitTombstone = GroupCoordinatorRecordHelpers.newOffsetCommitTombstoneRecord("group-id", "topic", 0);
+
+ @SuppressWarnings("unchecked")
+ ArgumentCaptor<List<CoordinatorRecord>> recordsCapture = ArgumentCaptor.forClass(List.class);
+
+ Group group = mock(Group.class);
+ when(group.shouldExpire(config)).thenReturn(false);
when(groupMetadataManager.groupIds()).thenReturn(Set.of("group-id"));
- when(groupMetadataManager.group("group-id")).thenReturn(streamsGroup);
- when(offsetMetadataManager.cleanupExpiredOffsets(eq("group-id"), any())).thenReturn(true);
+ when(groupMetadataManager.group("group-id")).thenReturn(group);
+ when(offsetMetadataManager.cleanupExpiredOffsets(eq("group-id"), recordsCapture.capture()))
+ .thenAnswer(invocation -> {
+ recordsCapture.getValue().add(offsetCommitTombstone);
+ return true;
+ });
- coordinator.cleanupGroupMetadata();
+ CoordinatorResult<Void, CoordinatorRecord> result = coordinator.cleanupGroupMetadata();
- verify(groupMetadataManager, times(1)).maybeDeleteGroup(eq("group-id"), any());
+ // Offset tombstone went through; group tombstone is deferred.
+ assertEquals(List.of(offsetCommitTombstone), result.records());
+ verify(offsetMetadataManager, times(1)).cleanupExpiredOffsets(eq("group-id"), any());
+ verify(groupMetadataManager, never()).maybeDeleteGroup(eq("group-id"), any());
}
@Test
- public void testCleanupGroupMetadataIgnoresStoredTopologyEpochWhenNoPluginConfigured() {
- // Plugin absent on this broker (operator unset / never set): even a streams group with
- // a non-default storedEpoch must NOT be deferred — no cleanup cycle is running to
- // clear it, and leaving the gate engaged would prevent natural expiration forever.
+ public void testCleanupGroupMetadataTombstonesWhenGroupAgreesToExpire() {
+ // Counterpart to the refuse test: when the group's shouldExpire(config) returns true
Review Comment:
"Counterpart to the refuse test" - not sure what it references. Maybe just omit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3481033598
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##########
@@ -911,22 +915,27 @@ private void recordPluginDeleteOutcome(int attempted, int errors) {
}
/**
- * Conditional metadata write that clears {@code StoredDescriptionTopologyEpoch} for
- * {@code groupId} only when the persisted value still equals {@code expectedStoredEpoch}.
- * Mismatches and missing groups are silently ignored by the shard-side method. Runtime
- * write failures (NOT_COORDINATOR etc.) are logged here and swallowed so a single failed
- * write does not poison the cycle's allOf — the next cycle will retry naturally because
- * the persisted storedEpoch is still non-default.
+ * Batched conditional metadata write that clears {@code StoredDescriptionTopologyEpoch}
+ * for every entry in {@code expectedStoredEpochByGroupId}, but only for the entries whose
+ * persisted value still equals the supplied epoch. Mismatches and missing groups are
+ * silently ignored by the shard-side method. All groups in the batch must hash to the same
+ * __consumer_offsets partition (the caller guarantees this — the eligibility scan is per
+ * partition). Runtime write failures (NOT_COORDINATOR etc.) are logged here and swallowed
+ * so a single failed write does not poison the cycle's allOf — the next cycle will retry
+ * naturally because the persisted storedEpoch is still non-default.
*/
- private CompletableFuture<Void> clearStoredDescriptionTopologyEpochAsync(String groupId, int expectedStoredEpoch) {
- return runtime.<Void>scheduleWriteOperation(
+ private CompletableFuture<Void> clearStoredDescriptionTopologyEpochBatchAsync(
+ Map<String, Integer> expectedStoredEpochByGroupId
+ ) {
+ TopicPartition tp = topicPartitionFor(expectedStoredEpochByGroupId.keySet().iterator().next());
Review Comment:
The batch's target partition is derived from `keySet().iterator().next()`, which throws `NoSuchElementException` on an empty map. It's safe today because the only caller guards with `if (toClear.isEmpty())`, but that coupling is implicit. Consider an early `if (expectedStoredEpochByGroupId.isEmpty()) return CompletableFuture.completedFuture(null);` here so the method is self-protecting and doesn't depend on the caller's guard staying in place.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##########
@@ -911,22 +915,27 @@ private void recordPluginDeleteOutcome(int attempted, int errors) {
}
/**
- * Conditional metadata write that clears {@code StoredDescriptionTopologyEpoch} for
- * {@code groupId} only when the persisted value still equals {@code expectedStoredEpoch}.
- * Mismatches and missing groups are silently ignored by the shard-side method. Runtime
- * write failures (NOT_COORDINATOR etc.) are logged here and swallowed so a single failed
- * write does not poison the cycle's allOf — the next cycle will retry naturally because
- * the persisted storedEpoch is still non-default.
+ * Batched conditional metadata write that clears {@code StoredDescriptionTopologyEpoch}
+ * for every entry in {@code expectedStoredEpochByGroupId}, but only for the entries whose
+ * persisted value still equals the supplied epoch. Mismatches and missing groups are
+ * silently ignored by the shard-side method. All groups in the batch must hash to the same
+ * __consumer_offsets partition (the caller guarantees this — the eligibility scan is per
+ * partition). Runtime write failures (NOT_COORDINATOR etc.) are logged here and swallowed
+ * so a single failed write does not poison the cycle's allOf — the next cycle will retry
+ * naturally because the persisted storedEpoch is still non-default.
*/
- private CompletableFuture<Void> clearStoredDescriptionTopologyEpochAsync(String groupId, int expectedStoredEpoch) {
- return runtime.<Void>scheduleWriteOperation(
+ private CompletableFuture<Void> clearStoredDescriptionTopologyEpochBatchAsync(
+ Map<String, Integer> expectedStoredEpochByGroupId
+ ) {
+ TopicPartition tp = topicPartitionFor(expectedStoredEpochByGroupId.keySet().iterator().next());
+ return runtime.scheduleWriteOperation(
"clear-stored-topology-epoch",
- topicPartitionFor(groupId),
- coordinator -> coordinator.clearStoredDescriptionTopologyEpoch(groupId, expectedStoredEpoch)
+ tp,
+ coordinator -> coordinator.clearStoredDescriptionTopologyEpochBatch(expectedStoredEpochByGroupId)
).handle((__, throwable) -> {
if (throwable != null) {
- log.warn("Failed to clear StoredDescriptionTopologyEpoch for group {}; the next cleanup cycle will retry.",
- groupId, throwable);
+ log.warn("Failed to clear StoredDescriptionTopologyEpoch for {} group(s) on partition {}; the next cleanup cycle will retry.",
Review Comment:
The batched failure log now reports `size()` + partition instead of the individual groupId, so a failed write no longer tells you which groups didn't get cleared. Given the next-cycle retry guarantee this is acceptable, but logging the group ids (here or at debug) would make a stuck group easier to diagnose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3491688658
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -1014,7 +1015,7 @@ public Map<String, Integer> listStreamsGroupsNeedingTopologyCleanup(long committ
if (!streamsGroup.isEmpty(committedOffset)) continue;
int storedEpoch = streamsGroup.storedDescriptionTopologyEpoch(committedOffset);
Review Comment:
Ah yes you are right I overlooked that. Yes, I don't see a very clean way around here, but I suppose 1 is the best we can do. In that case we could at least set
`shouldDeferGroupMetadataExpiration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Follow-up KAFKA-20696 [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #22676:
URL: https://github.com/apache/kafka/pull/22676#discussion_r3491641158
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorShard.java:
##########
@@ -1014,7 +1015,7 @@ public Map<String, Integer> listStreamsGroupsNeedingTopologyCleanup(long committ
if (!streamsGroup.isEmpty(committedOffset)) continue;
int storedEpoch = streamsGroup.storedDescriptionTopologyEpoch(committedOffset);
Review Comment:
Like the direction, but I think the literal fold introduces a circular dependency with `groupHasNoOffsets` simplification. Today's sweep order is:
```java
if (!group.shouldExpire()) return; // ShareGroup short-circuits here
boolean allOffsetsExpired = cleanupExpiredOffsets(...); // tombstones offsets
if (!allOffsetsExpired) return;
if (deferStreamsGroupTombstoneForPluginCleanup(...)) return; // streams gate only blocks the tombstone
maybeDeleteGroup(...);
```
A deferred streams group still runs `cleanupExpiredOffsets` — only the tombstone is skipped. That offset cleanup is the topology cycle's entry condition: it watches for `groupHasNoOffsets` (`offsetsByGroup` entry absent + no open txns) on the post-state of the regular sweep.
If we collapse the defer into `shouldExpire(config)`, the early return skips `cleanupExpiredOffsets` for any streams group with `storedDescriptionTopologyEpoch != -1`. That group's offsets never get tombstoned, the topology cycle's eligibility never trips, the stored epoch never gets cleared — and `shouldExpire(config)` stays false forever. The group can't unstick itself.
A few refinements that preserve the abstraction without the dependency cycle:
1. Two methods on Group: keep `shouldExpire()` for the "skip entire sweep" gate (ShareGroup's case), add a separate `shouldTombstoneAfterOffsetExpiration(config)` (or similar name) for streams' defer. The helper still goes away; the call site reads `if (!group.shouldExpire()) return; … if (!group.shouldTombstoneAfterOffsetExpiration(config)) return;`.
2. Reorder the sweep: always run `cleanupExpiredOffsets`, then check `shouldExpire(config)` for the tombstone decision. Cleaner in one sense but changes ShareGroup behaviour — its offsets would now be touched (no-op today since share groups don't hold consumer-style offsets, but worth confirming).
3. Something else you had in mind? Maybe the topology cycle should drive its own offset cleanup rather than depending on the regular sweep, which would decouple them entirely — but that's a bigger restructure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org