You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "lihaosky (via GitHub)" <gi...@apache.org> on 2023/08/07 23:08:48 UTC

[GitHub] [kafka] lihaosky opened a new pull request, #14164: KAFKA-15022: [8/N] more tests for HAAssignor

lihaosky opened a new pull request, #14164:
URL: https://github.com/apache/kafka/pull/14164

   ### Description
   1. More tests for `HighAvailabilityTaskAssignor`
   2. Remove null and optional check for `RackAwareTaskAssignor`
   
   ### Tests
   Update of existing tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287695148


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java:
##########
@@ -45,75 +81,162 @@ private static final class Harness {
         private final Map<UUID, ClientState> droppedClientStates;
         private final StringBuilder history = new StringBuilder();
 
+        public final Map<TaskId, Set<TopicPartition>> partitionsForTask;
+        public final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask;
+        public final Map<Subtopology, Set<TaskId>> tasksForTopicGroup;
+        public final Cluster fullMetadata;
+        public final Map<UUID, Map<String, Optional<String>>> racksForProcessConsumer;
+        public final InternalTopicManager internalTopicManager;
+
         private static Harness initializeCluster(final int numStatelessTasks,
                                                  final int numStatefulTasks,
-                                                 final int numNodes,
-                                                 final Supplier<Integer> partitionCountSupplier) {
+                                                 final int numClients,
+                                                 final Supplier<Integer> partitionCountSupplier,
+                                                 final int numNodes) {
             int subtopology = 0;
             final Set<TaskId> statelessTasks = new TreeSet<>();
             int remainingStatelessTasks = numStatelessTasks;
+            final List<Node> nodes = getRandomNodes(numNodes);
+            int nodeIndex = 0;
+            final Set<PartitionInfo> partitionInfoSet = new HashSet<>();
+            final Map<TaskId, Set<TopicPartition>> partitionsForTask = new HashMap<>();
+            final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask = new HashMap<>();
+            final Map<Subtopology, Set<TaskId>> tasksForTopicGroup = new HashMap<>();
+
             while (remainingStatelessTasks > 0) {
                 final int partitions = Math.min(remainingStatelessTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statelessTasks.add(new TaskId(subtopology, i));
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statelessTasks.add(taskId);
                     remainingStatelessTasks--;
+
+                    final Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
                 }
                 subtopology++;
             }
 
             final Map<TaskId, Long> statefulTaskEndOffsetSums = new TreeMap<>();
+            final Map<String, List<TopicPartitionInfo>> topicPartitionInfo = new HashMap<>();
+            final Set<String> changelogNames = new HashSet<>();
             int remainingStatefulTasks = numStatefulTasks;
             while (remainingStatefulTasks > 0) {
+                final String changelogTopicName = CHANGELOG_TOPIC_PREFIX + "_" + subtopology;
+                changelogNames.add(changelogTopicName);
                 final int partitions = Math.min(remainingStatefulTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statefulTaskEndOffsetSums.put(new TaskId(subtopology, i), 150000L);
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statefulTaskEndOffsetSums.put(taskId, 150000L);
                     remainingStatefulTasks--;
+
+                    Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    changelogPartitionsForTask.put(taskId, mkSet(new TopicPartition(changelogTopicName, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
+
+                    final Random random = new Random();

Review Comment:
   We should init random with a value that we log (to allow us to reproduce the same test)
   ```
   final long seed = System.currentTimeMillis():
   log.info(seed); // or just use println
   final Random random = new Random(seed);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287680080


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -272,27 +300,50 @@ private void createMockTaskManager(final Set<TaskId> activeTasks,
     // If mockCreateInternalTopics is true the internal topic manager will report that it had to create all internal
     // topics and we will skip the listOffsets request for these changelogs
     private MockInternalTopicManager overwriteInternalTopicManagerWithMock(final boolean mockCreateInternalTopics) {
-        final MockInternalTopicManager mockInternalTopicManager = new MockInternalTopicManager(
+        return overwriteInternalTopicManagerWithMock(mockCreateInternalTopics, null);
+    }
+
+    private MockInternalTopicManager overwriteInternalTopicManagerWithMock(final boolean mockCreateInternalTopics, final List<Map<String, List<TopicPartitionInfo>>> topicPartitionInfo) {
+        final MockInternalTopicManager mockInternalTopicManager = spy(new MockInternalTopicManager(
             time,
             new StreamsConfig(configProps()),
             mockClientSupplier.restoreConsumer,
             mockCreateInternalTopics
-        );
+        ));
+
+        if (topicPartitionInfo != null) {
+            lenient().when(mockInternalTopicManager.getTopicPartitionInfo(anySet())).thenAnswer(
+                i -> {

Review Comment:
   So `i` is the input parameter passed into `getTopicPartitionInfo` ? -- Can we find a better name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287697676


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java:
##########
@@ -45,75 +81,162 @@ private static final class Harness {
         private final Map<UUID, ClientState> droppedClientStates;
         private final StringBuilder history = new StringBuilder();
 
+        public final Map<TaskId, Set<TopicPartition>> partitionsForTask;
+        public final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask;
+        public final Map<Subtopology, Set<TaskId>> tasksForTopicGroup;
+        public final Cluster fullMetadata;
+        public final Map<UUID, Map<String, Optional<String>>> racksForProcessConsumer;
+        public final InternalTopicManager internalTopicManager;
+
         private static Harness initializeCluster(final int numStatelessTasks,
                                                  final int numStatefulTasks,
-                                                 final int numNodes,
-                                                 final Supplier<Integer> partitionCountSupplier) {
+                                                 final int numClients,
+                                                 final Supplier<Integer> partitionCountSupplier,
+                                                 final int numNodes) {
             int subtopology = 0;
             final Set<TaskId> statelessTasks = new TreeSet<>();
             int remainingStatelessTasks = numStatelessTasks;
+            final List<Node> nodes = getRandomNodes(numNodes);
+            int nodeIndex = 0;
+            final Set<PartitionInfo> partitionInfoSet = new HashSet<>();
+            final Map<TaskId, Set<TopicPartition>> partitionsForTask = new HashMap<>();
+            final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask = new HashMap<>();
+            final Map<Subtopology, Set<TaskId>> tasksForTopicGroup = new HashMap<>();
+
             while (remainingStatelessTasks > 0) {
                 final int partitions = Math.min(remainingStatelessTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statelessTasks.add(new TaskId(subtopology, i));
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statelessTasks.add(taskId);
                     remainingStatelessTasks--;
+
+                    final Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
                 }
                 subtopology++;
             }
 
             final Map<TaskId, Long> statefulTaskEndOffsetSums = new TreeMap<>();
+            final Map<String, List<TopicPartitionInfo>> topicPartitionInfo = new HashMap<>();
+            final Set<String> changelogNames = new HashSet<>();
             int remainingStatefulTasks = numStatefulTasks;
             while (remainingStatefulTasks > 0) {
+                final String changelogTopicName = CHANGELOG_TOPIC_PREFIX + "_" + subtopology;
+                changelogNames.add(changelogTopicName);
                 final int partitions = Math.min(remainingStatefulTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statefulTaskEndOffsetSums.put(new TaskId(subtopology, i), 150000L);
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statefulTaskEndOffsetSums.put(taskId, 150000L);
                     remainingStatefulTasks--;
+
+                    Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    changelogPartitionsForTask.put(taskId, mkSet(new TopicPartition(changelogTopicName, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
+
+                    final Random random = new Random();
+                    final int changelogNodeIndex = random.nextInt(nodes.size());
+                    replica = getRandomReplica(nodes, changelogNodeIndex);
+                    final TopicPartitionInfo info = new TopicPartitionInfo(i, replica[0], Arrays.asList(replica[0], replica[1]), Collections.emptyList());
+                    topicPartitionInfo.computeIfAbsent(changelogTopicName, tp -> new ArrayList<>()).add(info);
                 }
                 subtopology++;
             }
 
+            final MockTime time = new MockTime();
+            final StreamsConfig streamsConfig = new StreamsConfig(configProps(true));
+            final MockClientSupplier mockClientSupplier = new MockClientSupplier();
+            final MockInternalTopicManager mockInternalTopicManager = new MockInternalTopicManager(
+                time,
+                streamsConfig,
+                mockClientSupplier.restoreConsumer,
+                false
+            );
+            final InternalTopicManager spyTopicManager = spy(mockInternalTopicManager);
+            doReturn(topicPartitionInfo).when(spyTopicManager).getTopicPartitionInfo(changelogNames);
+
+            final Cluster cluster = new Cluster(
+                "cluster",
+                new HashSet<>(nodes),
+                partitionInfoSet,
+                Collections.emptySet(),
+                Collections.emptySet()
+            );
+
             final Map<UUID, ClientState> clientStates = new TreeMap<>();
-            for (int i = 0; i < numNodes; i++) {
+            final Map<UUID, Map<String, Optional<String>>> racksForProcessConsumer = new HashMap<>();
+            for (int i = 0; i < numClients; i++) {
                 final UUID uuid = uuidForInt(i);
                 clientStates.put(uuid, emptyInstance(uuid, statefulTaskEndOffsetSums));
+                final Random random = new Random();

Review Comment:
   as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287685324


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -1768,16 +1993,22 @@ public void shouldReturnInterleavedAssignmentWithUnrevokedPartitionsRemovedWhenN
         final Set<TaskId> allTasks = mkSet(TASK_0_0, TASK_0_1, TASK_0_2);
 
         subscriptions.put(CONSUMER_1,

Review Comment:
   nit: move `CONSUMER_1` to next line, too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287692186


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/AssignmentTestUtils.java:
##########
@@ -587,14 +587,18 @@ static List<Node> getRandomNodes(final int nodeSize) {
         return nodeList;
     }
 
+    static Node[] getRandomReplica(final List<Node> nodeList, final int index) {

Review Comment:
   Does not seem to be random actually?



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/AssignmentTestUtils.java:
##########
@@ -587,14 +587,18 @@ static List<Node> getRandomNodes(final int nodeSize) {
         return nodeList;
     }
 
+    static Node[] getRandomReplica(final List<Node> nodeList, final int index) {

Review Comment:
   Does not seem to be random actually?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287683621


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -1213,12 +1376,18 @@ public void testAssignWithInternalTopics() {
         final List<String> topics = asList("topic1", APPLICATION_ID + "-topicX");
         final Set<TaskId> allTasks = mkSet(TASK_0_0, TASK_0_1, TASK_0_2);
 
-        final MockInternalTopicManager internalTopicManager = configureDefault();
+        createDefaultMockTaskManager();
+        final List<Map<String, List<TopicPartitionInfo>>> changelogTopicPartitionInfo = getTopicPartitionInfo(4,

Review Comment:
   nit: move first parameter to next line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax merged pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #14164:
URL: https://github.com/apache/kafka/pull/14164


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287690983


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -2360,4 +2662,27 @@ private static Map<TaskId, Long> getTaskEndOffsetSums(final Collection<TaskId> a
         return allStatefulTasks.stream().collect(Collectors.toMap(t -> t, t -> Long.MAX_VALUE));
     }
 
+    private static List<Map<String, List<TopicPartitionInfo>>> getTopicPartitionInfo(final int replicaCount, final List<Set<String>> topicsList) {
+        final List<KeyValue<Node, List<Node>>> nodes = asList(
+            KeyValue.pair(NODE_0, asList(REPLICA_0)),
+            KeyValue.pair(NODE_1, asList(REPLICA_1)),
+            KeyValue.pair(NODE_2, asList(REPLICA_2)),
+            KeyValue.pair(NODE_3, asList(REPLICA_3))
+        );
+
+        final List<Map<String, List<TopicPartitionInfo>>> ret = new ArrayList<>();
+        for (final Set<String> topics : topicsList) {
+            ret.add(new HashMap<>());
+            for (final String topic : topics) {
+                final List<TopicPartitionInfo> topicPartitionInfoList = new ArrayList<>();
+                ret.get(ret.size() - 1).put(topic, topicPartitionInfoList);

Review Comment:
   Should we add a local variable instead of getting the map from the list?
   ```
   final Map topicToPartitionInfo = new HashMap();
   for (....) {
     ...
     topicToPartitionInfo.put(topic, topicPartitionInfoList);
   }
   ret.add(topicToPartitionInfo);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287685324


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -1768,16 +1993,22 @@ public void shouldReturnInterleavedAssignmentWithUnrevokedPartitionsRemovedWhenN
         final Set<TaskId> allTasks = mkSet(TASK_0_0, TASK_0_1, TASK_0_2);
 
         subscriptions.put(CONSUMER_1,

Review Comment:
   nit: move `CONSUMER_1` to next line, too (same below)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287683870


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -1241,12 +1410,18 @@ public void testAssignWithInternalTopicThatsSourceIsAnotherInternalTopic() {
         final List<String> topics = asList("topic1", APPLICATION_ID + "-topicX", APPLICATION_ID + "-topicZ");
         final Set<TaskId> allTasks = mkSet(TASK_0_0, TASK_0_1, TASK_0_2);
 
-        final MockInternalTopicManager internalTopicManager = configureDefault();
+        createDefaultMockTaskManager();
+        final List<Map<String, List<TopicPartitionInfo>>> changelogTopicPartitionInfo = getTopicPartitionInfo(4,

Review Comment:
   as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287690983


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -2360,4 +2662,27 @@ private static Map<TaskId, Long> getTaskEndOffsetSums(final Collection<TaskId> a
         return allStatefulTasks.stream().collect(Collectors.toMap(t -> t, t -> Long.MAX_VALUE));
     }
 
+    private static List<Map<String, List<TopicPartitionInfo>>> getTopicPartitionInfo(final int replicaCount, final List<Set<String>> topicsList) {
+        final List<KeyValue<Node, List<Node>>> nodes = asList(
+            KeyValue.pair(NODE_0, asList(REPLICA_0)),
+            KeyValue.pair(NODE_1, asList(REPLICA_1)),
+            KeyValue.pair(NODE_2, asList(REPLICA_2)),
+            KeyValue.pair(NODE_3, asList(REPLICA_3))
+        );
+
+        final List<Map<String, List<TopicPartitionInfo>>> ret = new ArrayList<>();
+        for (final Set<String> topics : topicsList) {
+            ret.add(new HashMap<>());
+            for (final String topic : topics) {
+                final List<TopicPartitionInfo> topicPartitionInfoList = new ArrayList<>();
+                ret.get(ret.size() - 1).put(topic, topicPartitionInfoList);

Review Comment:
   Should we add a local variable instead of getting the map from the list? -- Similar to `topicPartitionInfoList`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287681335


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -272,27 +300,50 @@ private void createMockTaskManager(final Set<TaskId> activeTasks,
     // If mockCreateInternalTopics is true the internal topic manager will report that it had to create all internal
     // topics and we will skip the listOffsets request for these changelogs
     private MockInternalTopicManager overwriteInternalTopicManagerWithMock(final boolean mockCreateInternalTopics) {
-        final MockInternalTopicManager mockInternalTopicManager = new MockInternalTopicManager(
+        return overwriteInternalTopicManagerWithMock(mockCreateInternalTopics, null);
+    }
+
+    private MockInternalTopicManager overwriteInternalTopicManagerWithMock(final boolean mockCreateInternalTopics, final List<Map<String, List<TopicPartitionInfo>>> topicPartitionInfo) {
+        final MockInternalTopicManager mockInternalTopicManager = spy(new MockInternalTopicManager(
             time,
             new StreamsConfig(configProps()),
             mockClientSupplier.restoreConsumer,
             mockCreateInternalTopics
-        );
+        ));
+
+        if (topicPartitionInfo != null) {
+            lenient().when(mockInternalTopicManager.getTopicPartitionInfo(anySet())).thenAnswer(
+                i -> {
+                    final Set<String> topics = i.getArgument(0);
+                    for (final Map<String, List<TopicPartitionInfo>> tp : topicPartitionInfo) {
+                        if (topics.equals(tp.keySet())) {
+                            return tp;
+                        }
+                    }
+                    return null;

Review Comment:
   Is `null` right here? Or should it be an empty `Map` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287675546


##########
streams/src/test/java/org/apache/kafka/streams/integration/HighAvailabilityTaskAssignorIntegrationTest.java:
##########
@@ -300,7 +329,9 @@ private static Properties streamsProperties(final String appId,
                 // Increasing the number of threads to ensure that a rebalance happens each time a consumer sends a rejoin (KAFKA-10455)
                 mkEntry(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 40),
                 mkEntry(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName()),
-                mkEntry(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName())
+                mkEntry(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName()),
+                mkEntry(CommonClientConfigs.CLIENT_RACK_CONFIG, AssignmentTestUtils.RACK_0),

Review Comment:
   Why is this hard-coded? Should we use different racks for different KS instances?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287682340


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -809,14 +888,37 @@ public void testAssignWithStates() {
                    APPLICATION_ID + "-store3-changelog"),
             asList(3, 3, 3))
         );
-        configureDefault();
+
+        createDefaultMockTaskManager();
+        final List<Map<String, List<TopicPartitionInfo>>> changelogTopicPartitionInfo = getTopicPartitionInfo(3,

Review Comment:
   nit: move first parameter `3,` into it's own line below (hard to read the code otherwise)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287697676


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java:
##########
@@ -45,75 +81,162 @@ private static final class Harness {
         private final Map<UUID, ClientState> droppedClientStates;
         private final StringBuilder history = new StringBuilder();
 
+        public final Map<TaskId, Set<TopicPartition>> partitionsForTask;
+        public final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask;
+        public final Map<Subtopology, Set<TaskId>> tasksForTopicGroup;
+        public final Cluster fullMetadata;
+        public final Map<UUID, Map<String, Optional<String>>> racksForProcessConsumer;
+        public final InternalTopicManager internalTopicManager;
+
         private static Harness initializeCluster(final int numStatelessTasks,
                                                  final int numStatefulTasks,
-                                                 final int numNodes,
-                                                 final Supplier<Integer> partitionCountSupplier) {
+                                                 final int numClients,
+                                                 final Supplier<Integer> partitionCountSupplier,
+                                                 final int numNodes) {
             int subtopology = 0;
             final Set<TaskId> statelessTasks = new TreeSet<>();
             int remainingStatelessTasks = numStatelessTasks;
+            final List<Node> nodes = getRandomNodes(numNodes);
+            int nodeIndex = 0;
+            final Set<PartitionInfo> partitionInfoSet = new HashSet<>();
+            final Map<TaskId, Set<TopicPartition>> partitionsForTask = new HashMap<>();
+            final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask = new HashMap<>();
+            final Map<Subtopology, Set<TaskId>> tasksForTopicGroup = new HashMap<>();
+
             while (remainingStatelessTasks > 0) {
                 final int partitions = Math.min(remainingStatelessTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statelessTasks.add(new TaskId(subtopology, i));
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statelessTasks.add(taskId);
                     remainingStatelessTasks--;
+
+                    final Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
                 }
                 subtopology++;
             }
 
             final Map<TaskId, Long> statefulTaskEndOffsetSums = new TreeMap<>();
+            final Map<String, List<TopicPartitionInfo>> topicPartitionInfo = new HashMap<>();
+            final Set<String> changelogNames = new HashSet<>();
             int remainingStatefulTasks = numStatefulTasks;
             while (remainingStatefulTasks > 0) {
+                final String changelogTopicName = CHANGELOG_TOPIC_PREFIX + "_" + subtopology;
+                changelogNames.add(changelogTopicName);
                 final int partitions = Math.min(remainingStatefulTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statefulTaskEndOffsetSums.put(new TaskId(subtopology, i), 150000L);
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statefulTaskEndOffsetSums.put(taskId, 150000L);
                     remainingStatefulTasks--;
+
+                    Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    changelogPartitionsForTask.put(taskId, mkSet(new TopicPartition(changelogTopicName, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
+
+                    final Random random = new Random();
+                    final int changelogNodeIndex = random.nextInt(nodes.size());
+                    replica = getRandomReplica(nodes, changelogNodeIndex);
+                    final TopicPartitionInfo info = new TopicPartitionInfo(i, replica[0], Arrays.asList(replica[0], replica[1]), Collections.emptyList());
+                    topicPartitionInfo.computeIfAbsent(changelogTopicName, tp -> new ArrayList<>()).add(info);
                 }
                 subtopology++;
             }
 
+            final MockTime time = new MockTime();
+            final StreamsConfig streamsConfig = new StreamsConfig(configProps(true));
+            final MockClientSupplier mockClientSupplier = new MockClientSupplier();
+            final MockInternalTopicManager mockInternalTopicManager = new MockInternalTopicManager(
+                time,
+                streamsConfig,
+                mockClientSupplier.restoreConsumer,
+                false
+            );
+            final InternalTopicManager spyTopicManager = spy(mockInternalTopicManager);
+            doReturn(topicPartitionInfo).when(spyTopicManager).getTopicPartitionInfo(changelogNames);
+
+            final Cluster cluster = new Cluster(
+                "cluster",
+                new HashSet<>(nodes),
+                partitionInfoSet,
+                Collections.emptySet(),
+                Collections.emptySet()
+            );
+
             final Map<UUID, ClientState> clientStates = new TreeMap<>();
-            for (int i = 0; i < numNodes; i++) {
+            final Map<UUID, Map<String, Optional<String>>> racksForProcessConsumer = new HashMap<>();
+            for (int i = 0; i < numClients; i++) {
                 final UUID uuid = uuidForInt(i);
                 clientStates.put(uuid, emptyInstance(uuid, statefulTaskEndOffsetSums));
+                final Random random = new Random();

Review Comment:
   as above: maybe setup `Random` inside `@BeforeTest` instead and share?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287701495


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java:
##########
@@ -227,15 +350,38 @@ private void formatClientStates(final boolean printUnassigned) {
         }
     }
 
+    @Parameter
+    public boolean enableRackAwareTaskAssignor;
+
+    private String rackAwareStrategy = StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_NONE;
+
+    @Before
+    public void setUp() {
+        if (enableRackAwareTaskAssignor) {
+            rackAwareStrategy = StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_MIN_TRAFFIC;
+        }
+    }
+
+    @Parameterized.Parameters(name = "enableRackAwareTaskAssignor={0}")
+    public static Collection<Object[]> getParamStoreType() {
+        return asList(new Object[][] {
+            {true},
+            {false}
+        });
+    }
+
     @Test
     public void staticAssignmentShouldConvergeWithTheFirstAssignment() {
         final AssignmentConfigs configs = new AssignmentConfigs(100L,
                                                                 2,
                                                                 0,
                                                                 60_000L,
-                                                                EMPTY_RACK_AWARE_ASSIGNMENT_TAGS);
+                                                                EMPTY_RACK_AWARE_ASSIGNMENT_TAGS,
+                                    null,
+                                null,

Review Comment:
   fix indention



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java:
##########
@@ -248,21 +394,29 @@ public void assignmentShouldConvergeAfterAddingNode() {
         final int numStatefulTasks = 11;
         final int maxWarmupReplicas = 2;
         final int numStandbyReplicas = 0;
+        final int numNodes = 10;
 
         final AssignmentConfigs configs = new AssignmentConfigs(100L,
                                                                 maxWarmupReplicas,
                                                                 numStandbyReplicas,
                                                                 60_000L,
-                                                                EMPTY_RACK_AWARE_ASSIGNMENT_TAGS);
+                                                                EMPTY_RACK_AWARE_ASSIGNMENT_TAGS,
+                                    null,
+                                null,

Review Comment:
   fix indention



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287695148


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/assignment/TaskAssignorConvergenceTest.java:
##########
@@ -45,75 +81,162 @@ private static final class Harness {
         private final Map<UUID, ClientState> droppedClientStates;
         private final StringBuilder history = new StringBuilder();
 
+        public final Map<TaskId, Set<TopicPartition>> partitionsForTask;
+        public final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask;
+        public final Map<Subtopology, Set<TaskId>> tasksForTopicGroup;
+        public final Cluster fullMetadata;
+        public final Map<UUID, Map<String, Optional<String>>> racksForProcessConsumer;
+        public final InternalTopicManager internalTopicManager;
+
         private static Harness initializeCluster(final int numStatelessTasks,
                                                  final int numStatefulTasks,
-                                                 final int numNodes,
-                                                 final Supplier<Integer> partitionCountSupplier) {
+                                                 final int numClients,
+                                                 final Supplier<Integer> partitionCountSupplier,
+                                                 final int numNodes) {
             int subtopology = 0;
             final Set<TaskId> statelessTasks = new TreeSet<>();
             int remainingStatelessTasks = numStatelessTasks;
+            final List<Node> nodes = getRandomNodes(numNodes);
+            int nodeIndex = 0;
+            final Set<PartitionInfo> partitionInfoSet = new HashSet<>();
+            final Map<TaskId, Set<TopicPartition>> partitionsForTask = new HashMap<>();
+            final Map<TaskId, Set<TopicPartition>> changelogPartitionsForTask = new HashMap<>();
+            final Map<Subtopology, Set<TaskId>> tasksForTopicGroup = new HashMap<>();
+
             while (remainingStatelessTasks > 0) {
                 final int partitions = Math.min(remainingStatelessTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statelessTasks.add(new TaskId(subtopology, i));
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statelessTasks.add(taskId);
                     remainingStatelessTasks--;
+
+                    final Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
                 }
                 subtopology++;
             }
 
             final Map<TaskId, Long> statefulTaskEndOffsetSums = new TreeMap<>();
+            final Map<String, List<TopicPartitionInfo>> topicPartitionInfo = new HashMap<>();
+            final Set<String> changelogNames = new HashSet<>();
             int remainingStatefulTasks = numStatefulTasks;
             while (remainingStatefulTasks > 0) {
+                final String changelogTopicName = CHANGELOG_TOPIC_PREFIX + "_" + subtopology;
+                changelogNames.add(changelogTopicName);
                 final int partitions = Math.min(remainingStatefulTasks, partitionCountSupplier.get());
                 for (int i = 0; i < partitions; i++) {
-                    statefulTaskEndOffsetSums.put(new TaskId(subtopology, i), 150000L);
+                    final TaskId taskId = new TaskId(subtopology, i);
+                    statefulTaskEndOffsetSums.put(taskId, 150000L);
                     remainingStatefulTasks--;
+
+                    Node[] replica = getRandomReplica(nodes, nodeIndex);
+                    partitionInfoSet.add(new PartitionInfo(TOPIC_PREFIX + "_" + subtopology, i, replica[0], replica, replica));
+                    nodeIndex++;
+
+                    partitionsForTask.put(taskId, mkSet(new TopicPartition(TOPIC_PREFIX + "_" + subtopology, i)));
+                    changelogPartitionsForTask.put(taskId, mkSet(new TopicPartition(changelogTopicName, i)));
+                    tasksForTopicGroup.computeIfAbsent(new Subtopology(subtopology, null), k -> new HashSet<>()).add(taskId);
+
+                    final Random random = new Random();

Review Comment:
   We should init `Random` with a value that we log (to allow us to reproduce the same test)
   ```
   final long seed = System.currentTimeMillis():
   log.info(seed); // or just use println
   final Random random = new Random(seed);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287689072


##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -2360,4 +2662,27 @@ private static Map<TaskId, Long> getTaskEndOffsetSums(final Collection<TaskId> a
         return allStatefulTasks.stream().collect(Collectors.toMap(t -> t, t -> Long.MAX_VALUE));
     }
 
+    private static List<Map<String, List<TopicPartitionInfo>>> getTopicPartitionInfo(final int replicaCount, final List<Set<String>> topicsList) {
+        final List<KeyValue<Node, List<Node>>> nodes = asList(
+            KeyValue.pair(NODE_0, asList(REPLICA_0)),
+            KeyValue.pair(NODE_1, asList(REPLICA_1)),
+            KeyValue.pair(NODE_2, asList(REPLICA_2)),
+            KeyValue.pair(NODE_3, asList(REPLICA_3))
+        );
+
+        final List<Map<String, List<TopicPartitionInfo>>> ret = new ArrayList<>();

Review Comment:
   `ret` is not a good variable name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mjsax commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287676811


##########
streams/src/test/java/org/apache/kafka/streams/integration/utils/KafkaEmbedded.java:
##########
@@ -91,6 +91,7 @@ private Properties effectiveConfigFrom(final Properties initialConfig) {
         effectiveConfig.put(KafkaConfig.MessageMaxBytesProp(), 1000000);
         effectiveConfig.put(KafkaConfig.ControlledShutdownEnableProp(), true);
         effectiveConfig.put(KafkaConfig.ZkSessionTimeoutMsProp(), 10000);
+        effectiveConfig.put(KafkaConfig.RackProp(), "rack0");

Review Comment:
   why do we need this overwrite?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] lihaosky commented on a diff in pull request #14164: KAFKA-15022: [8/N] more tests for HAAssignor

Posted by "lihaosky (via GitHub)" <gi...@apache.org>.
lihaosky commented on code in PR #14164:
URL: https://github.com/apache/kafka/pull/14164#discussion_r1287753972


##########
streams/src/test/java/org/apache/kafka/streams/integration/utils/KafkaEmbedded.java:
##########
@@ -91,6 +91,7 @@ private Properties effectiveConfigFrom(final Properties initialConfig) {
         effectiveConfig.put(KafkaConfig.MessageMaxBytesProp(), 1000000);
         effectiveConfig.put(KafkaConfig.ControlledShutdownEnableProp(), true);
         effectiveConfig.put(KafkaConfig.ZkSessionTimeoutMsProp(), 10000);
+        effectiveConfig.put(KafkaConfig.RackProp(), "rack0");

Review Comment:
   Good catch. Might be some left over change. Let me delete it



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java:
##########
@@ -272,27 +300,50 @@ private void createMockTaskManager(final Set<TaskId> activeTasks,
     // If mockCreateInternalTopics is true the internal topic manager will report that it had to create all internal
     // topics and we will skip the listOffsets request for these changelogs
     private MockInternalTopicManager overwriteInternalTopicManagerWithMock(final boolean mockCreateInternalTopics) {
-        final MockInternalTopicManager mockInternalTopicManager = new MockInternalTopicManager(
+        return overwriteInternalTopicManagerWithMock(mockCreateInternalTopics, null);
+    }
+
+    private MockInternalTopicManager overwriteInternalTopicManagerWithMock(final boolean mockCreateInternalTopics, final List<Map<String, List<TopicPartitionInfo>>> topicPartitionInfo) {
+        final MockInternalTopicManager mockInternalTopicManager = spy(new MockInternalTopicManager(
             time,
             new StreamsConfig(configProps()),
             mockClientSupplier.restoreConsumer,
             mockCreateInternalTopics
-        );
+        ));
+
+        if (topicPartitionInfo != null) {
+            lenient().when(mockInternalTopicManager.getTopicPartitionInfo(anySet())).thenAnswer(
+                i -> {
+                    final Set<String> topics = i.getArgument(0);
+                    for (final Map<String, List<TopicPartitionInfo>> tp : topicPartitionInfo) {
+                        if (topics.equals(tp.keySet())) {
+                            return tp;
+                        }
+                    }
+                    return null;

Review Comment:
   We can return empty Map.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org