[GitHub] [kafka] C0urante commented on a diff in pull request #13446: KAFKA-14837, KAFKA-14842: Ignore groups that do not have offsets for filtered topics in MirrorCheckpointConnector

["C0urante (via GitHub)" <gi...@apache.org>]

C0urante commented on code in PR #13446:
URL: https://github.com/apache/kafka/pull/13446#discussion_r1244478202


##########
connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointConnector.java:
##########
@@ -150,10 +156,31 @@ private void loadInitialConsumerGroups()
 
     List<String> findConsumerGroups()
             throws InterruptedException, ExecutionException {
-        return listConsumerGroups().stream()
+        List<String> filteredGroups = listConsumerGroups().stream()
                 .map(ConsumerGroupListing::groupId)
-                .filter(this::shouldReplicate)
+                .filter(this::shouldReplicateByGroupFilter)
                 .collect(Collectors.toList());
+
+        List<String> checkpointGroups = new LinkedList<>();
+        List<String> irrelevantGroups = new LinkedList<>();
+
+        for (String group : filteredGroups) {
+            Set<String> consumedTopics = listConsumerGroupOffsets(group).keySet().stream()
+                    .map(TopicPartition::topic)
+                    .filter(this::shouldReplicateByTopicFilter)

Review Comment:
   I don't believe MM2 ever supported that. From [KIP-382](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-InternalTopics), regarding checkpoints (emphasis mine):
   
   > The connector will periodically query the source cluster for all committed offsets from all consumer groups, **filter for those topics being replicated**, and emit a message to a topic
   
   IIUC, MM2 also performs offset syncing based on the contents of the offset syncs topic, which is only populated by the source connector (i.e., the connector that replicates topics).
   
   @blacktooth have you experienced otherwise?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org