[GitHub] [kafka] zhaohaidao commented on a change in pull request #8550: KAFKA-9850 Move KStream#repartition operator validation during Topolo…

[GitBox <gi...@apache.org>]

zhaohaidao commented on a change in pull request #8550:
URL: https://github.com/apache/kafka/pull/8550#discussion_r415334757



##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopologyBuilder.java
##########
@@ -633,6 +634,45 @@ public final void copartitionSources(final Collection<String> sourceNodes) {
         copartitionSourceGroups.add(Collections.unmodifiableSet(new HashSet<>(sourceNodes)));
     }
 
+    public void validateCoPartition() {
+        final List<Set<String>> copartitionGroups =
+                copartitionSourceGroups
+                        .stream()
+                        .map(sourceGroup -> sourceGroup
+                                .stream()
+                                .flatMap(node -> nodeToSourceTopics.get(node).stream())
+                                .collect(Collectors.toSet())
+                        ).collect(Collectors.toList());
+        for (final Set<String> coPartition : copartitionGroups) {
+            final Map<String, InternalTopicProperties> coPartitionProperties = new HashMap<>();
+            internalTopicNamesWithProperties.forEach((topic, prop) -> {
+                if (coPartition.contains(topic) && prop.getNumberOfPartitions().isPresent()) {
+                    coPartitionProperties.put(topic, prop);
+                }
+            });
+            if (coPartition.size() == coPartitionProperties.size()) {

Review comment:
       It's my pleasure.
   It means that not all input topics have correspond internal topic if coPartition.size() != coPartitionProperties.size(), if not equal is true, we can just skip this validation. You can see the original validation in CopartitionedTopicsEnforcer#enforce
   ```
   if (copartitionGroup.equals(repartitionTopicConfigs.keySet())) {
       ...
       validateAndGetNumOfPartitions
       ...
   }
   ```
   If some of input topics don't have repartition operation, their internal topic partition number can be deducted by others which have repartition operation. You can see KStreamRepartitionIntegrationTest#shouldDeductNumberOfPartitionsFromRepartitionOperation for more details.
   
    So we can skip this validation if coPartition.size() != coPartitionProperties.size()




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org