You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "rreddy-22 (via GitHub)" <gi...@apache.org> on 2023/09/20 23:24:59 UTC
[GitHub] [kafka] rreddy-22 opened a new pull request, #14416: KAFKA 14515: Latest Optimized Uniform Rack Aware Assignor
rreddy-22 opened a new pull request, #14416:
URL: https://github.com/apache/kafka/pull/14416
This module contains code for the implementation of the Optimized Uniform Rack Aware Assignor.
The files added are:
Uniform Assignor: Parent class with common functionality, general and optimized assignors extend this class.
Abstract Uniform Assignment Builder: Abstract class for assignment builder with generic methods and rack info class.
Optimized Uniform Assignment Builder: Used when all the topic subscriptions are equal for all members in the consumer group.
General Uniform Assignment Builder: Used in cases where topic subscriptions are different. Not fully implemented in this module. The file contains the basics required for the Uniform Assignor to run.
Tests added to ensure that first time assignments and reassignments take place as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340500692
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
Review Comment:
We're not iterating through the whole list though? For each member you get a list of topicIdPartitions that are valid. Then based on the quotas we just pick the first few and add it to the target assignment and also remove it from unassigned partitions. We're iterating through the retainedPartition count which is the number of partitions we want to retain, not the entire partitions count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1378164028
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * List of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> racksPerPartition;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
Review Comment:
changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340488260
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignment,
Review Comment:
David wanted it to be more generic so that we can reuse it, I can make it memberAssignments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #14416:
URL: https://github.com/apache/kafka/pull/14416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340525450
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
Review Comment:
converted to set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340482277
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
Review Comment:
Written like that cause we provide the topic and the partition belonging to the topic, so the topic's partition is added as opposed to the topicIdPartition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jeffkbkim commented on a diff in pull request #14416: KAFKA 14515: Latest Optimized Uniform Rack Aware Assignor
Posted by "jeffkbkim (via GitHub)" <gi...@apache.org>.
jeffkbkim commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1339390399
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignment,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignment.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * Number of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> partitionRacks;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ partitionRacks = Collections.emptyMap();
+ } else {
+ partitionRacks = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions.forEach(tp -> {
+ Set<String> racks = subscribedTopicDescriber.racksForPartition(tp.topicId(), tp.partition());
+ partitionRacks.put(tp, racks);
+ if (!racks.isEmpty()) allPartitionRacks.addAll(racks);
+ });
+ }
+
+ if (useRackAwareAssignment(membersByRack.keySet(), allPartitionRacks, partitionRacks)) {
Review Comment:
didn't we agree to change the name? allPartitionRacks and partitionRacks are confusing to differentiate. let's use racksByPartition or racksPerPartition and unify that inside `useRackAwareAssignment`
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignment,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignment.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * Number of members with the same rack as the partition.
Review Comment:
List of members
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
Review Comment:
nit: current assignment
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignment,
Review Comment:
either targetAssignment or memberAssignments makes more sense to me
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/UniformAssignor.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The Uniform Assignor distributes topic partitions among group members for a
+ * balanced and potentially rack aware assignment.
+ * The assignor employs two different strategies based on the nature of topic
+ * subscriptions across the group members:
+ * <ul>
+ * <li>
+ * <b> Optimized Uniform Assignment Builder: </b> This strategy is used when all members have subscribed
+ * to the same set of topics.
+ * </li>
+ * <li>
+ * <b> General Uniform Assignment Builder: </b> This strategy is used when members have varied topic
+ * subscriptions.
+ * </li>
+ * </ul>
+ *
+ * The appropriate strategy is automatically chosen based on the current members' topic subscriptions.
+ *
+ * @see OptimizedUniformAssignmentBuilder
+ * @see GeneralUniformAssignmentBuilder
+ */
+public class UniformAssignor implements PartitionAssignor {
+ private static final Logger LOG = LoggerFactory.getLogger(UniformAssignor.class);
+ public static final String UNIFORM_ASSIGNOR_NAME = "uniform";
+
+ @Override
+ public String name() {
+ return UNIFORM_ASSIGNOR_NAME;
+ }
+
+ /**
+ * Perform the group assignment given the current members and
+ * topics metadata.
+ *
+ * @param assignmentSpec The assignment specification that included member metadata.
+ * @param subscribedTopicDescriber The topic and cluster metadata describer {@link SubscribedTopicDescriber}.
+ * @return The new target assignment for the group.
+ */
+ @Override
+ public GroupAssignment assign(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) throws PartitionAssignorException {
+ AbstractUniformAssignmentBuilder assignmentBuilder;
+
+ if (assignmentSpec.members().isEmpty())
+ return new GroupAssignment(Collections.emptyMap());
+
+ if (allSubscriptionsEqual(assignmentSpec.members())) {
+ LOG.debug("Detected that all members are subscribed to the same set of topics, invoking the "
+ + "optimized assignment algorithm");
+ assignmentBuilder = new OptimizedUniformAssignmentBuilder(assignmentSpec, subscribedTopicDescriber);
+ } else {
+ LOG.debug("Detected that the members are subscribed to different sets of topics, invoking the "
+ + "general assignment algorithm");
+ assignmentBuilder = new GeneralUniformAssignmentBuilder();
+ }
+
+ return assignmentBuilder.buildAssignment();
+ }
+
+ /**
+ * Determines if all members are subscribed to the same list of topic Ids.
+ *
+ * @param members Members mapped to their respective {@code AssignmentMemberSpec}.
+ * @return true if all members have the same subscription list of topic Ids,
+ * false otherwise.
+ */
+ private boolean allSubscriptionsEqual(Map<String, AssignmentMemberSpec> members) {
+ Set<Uuid> firstSubscriptionSet = new HashSet<>(members.values().iterator().next().subscribedTopicIds());
+ for (AssignmentMemberSpec memberSpec : members.values()) {
+ if (!firstSubscriptionSet.containsAll(memberSpec.subscribedTopicIds())) {
+ return false;
+ }
+ }
+ return true;
+ }
+}
Review Comment:
newline
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentAssignment The partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment,
Review Comment:
can remove comma
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
Review Comment:
Adds the topic partition
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
Review Comment:
> assign the next partition as well
this is assuming that the next partition is rack matching right?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignment,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignment.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * Number of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> partitionRacks;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ partitionRacks = Collections.emptyMap();
+ } else {
+ partitionRacks = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions.forEach(tp -> {
+ Set<String> racks = subscribedTopicDescriber.racksForPartition(tp.topicId(), tp.partition());
+ partitionRacks.put(tp, racks);
+ if (!racks.isEmpty()) allPartitionRacks.addAll(racks);
+ });
+ }
+
+ if (useRackAwareAssignment(membersByRack.keySet(), allPartitionRacks, partitionRacks)) {
+ this.memberRacks = new HashMap<>(assignmentSpec.members().size());
+ membersByRack.forEach((rack, rackMembers) -> rackMembers.forEach(c -> memberRacks.put(c, rack)));
+ this.partitionRacks = partitionRacks;
+ useRackStrategy = true;
+ } else {
+ this.memberRacks = Collections.emptyMap();
+ this.partitionRacks = Collections.emptyMap();
+ useRackStrategy = false;
+ }
+
+ this.membersWithSameRackAsPartition = partitionRacks.entrySet().stream()
+ .collect(Collectors.toMap(
+ Map.Entry::getKey,
+ entry -> entry.getValue().stream()
+ .flatMap(rack -> membersByRack.getOrDefault(rack, Collections.emptyList()).stream())
+ .distinct() // Ensure that there are no duplicate members
+ .collect(Collectors.toList())
+ ));
+ }
+
+ /**
+ * Determines if there's a mismatch between the member's rack and the partition's replica racks.
+ *
+ * <p> Racks are considered mismatched under the following conditions: (returns {@code true}):
+ * <ul>
+ * <li> Member lacks an associated rack. </li>
+ * <li> Partition lacks associated replica racks. </li>
+ * <li> Member's rack isn't among the partition's replica racks. </li>
+ * </ul>
+ *
+ * @param memberId The member Id.
+ * @param tp The topic partition.
+ * @return {@code true} for a mismatch; {@code false} if member and partition racks exist and align.
+ */
+ protected boolean racksMismatch(String memberId, TopicIdPartition tp) {
+ String memberRack = memberRacks.get(memberId);
+ Set<String> replicaRacks = partitionRacks.get(tp);
+ return memberRack == null || (replicaRacks == null || !replicaRacks.contains(memberRack));
+ }
+
+ /**
+ * Sort partitions in ascending order by number of members with matching racks.
+ *
+ * @param partitions The list of partitions to be sorted.
+ * @return A sorted list of partitions with potential members in the same rack.
+ */
+ protected List<TopicIdPartition> sortPartitionsByRackMembers(List<TopicIdPartition> partitions) {
+ if (membersWithSameRackAsPartition.isEmpty())
+ return partitions;
+
+ return partitions.stream()
+ .filter(tp -> {
+ int count = membersWithSameRackAsPartition.get(tp).size();
+ return count > 0;
+ })
+ .sorted(Comparator.comparing(tp ->
+ membersWithSameRackAsPartition.getOrDefault(tp, Collections.emptyList()).size()
+ ))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * @return List of members with the same rack as any of the provided partition's replicas.
+ * Members are sorted in ascending order of number of partitions in the assignment.
+ */
+ protected List<String> getSortedMembersWithMatchingRack(
+ TopicIdPartition topicIdPartition,
+ Map<String, MemberAssignment> assignment
+ ) {
+ List<String> membersList = membersWithSameRackAsPartition.getOrDefault(
+ topicIdPartition,
+ Collections.emptyList()
+ );
+
+ // Sort the list based on the size of each member's assignment.
+ membersList.sort((member1, member2) -> {
+ int sum1 = assignment.get(member1).targetPartitions().values().stream().mapToInt(Set::size).sum();
+ int sum2 = assignment.get(member2).targetPartitions().values().stream().mapToInt(Set::size).sum();
+
+ return Integer.compare(sum1, sum2);
+ });
+
+ return membersList;
+ }
+
+ @Override
+ public String toString() {
+ return "RackInfo(" +
+ "memberRacks=" + memberRacks +
+ ", partitionRacks=" + partitionRacks +
+ ")";
+ }
+ }
+}
Review Comment:
newline
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneralUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(GeneralUniformAssignmentBuilder.class);
+
+ @Override
+ protected GroupAssignment buildAssignment() {
+ return null;
+ }
+}
Review Comment:
newline
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
Review Comment:
i think we can inline this for L213, L215, L216
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
Review Comment:
this is unfortunate as we are removing from a list. we're iterating over the entire topic partitions list for each member, num_partition times.
do we need the sorted topic partition list before we do these removals (including L237)? maybe we can keep it in a set, remove, then convert that into a sorted list.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadata.java:
##########
@@ -68,12 +69,12 @@ public int numPartitions(Uuid topicId) {
* @param topicId Uuid corresponding to the partition's topic.
* @param partition Partition Id within the topic.
* @return The set of racks corresponding to the replicas of the topics partition.
- * If the topic Id does not exist, an empty set is returned
+ * If the topic Id does not exist or no partition rack information is available, an empty set is returned.
*/
@Override
public Set<String> racksForPartition(Uuid topicId, int partition) {
TopicMetadata topic = this.topicMetadata.get(topicId);
- return topic == null ? Collections.emptySet() : topic.partitionRacks().get(partition);
+ return (topic == null) ? Collections.emptySet() : topic.partitionRacks().getOrDefault(partition, Collections.emptySet());
Review Comment:
the parentheses around `topic == null` seems a bit too much
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadata.java:
##########
@@ -30,6 +30,7 @@
* topic and partition metadata for the topics that the consumer group is subscribed to.
*/
public class SubscribedTopicMetadata implements SubscribedTopicDescriber {
+
Review Comment:
do we need this?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
Review Comment:
i'm wondering whether we can differentiate the two:
* target assignment: the target [memberId, member assignment]
* member assignment: the member's assignment.
so, i propose to make member assignment plural and target assignment singular.
this would then be `validCurrentMemberAssignment` and all usages of multiple member assignments would be plural. what do you think?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/common/TopicIdPartition.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.common;
+
+import org.apache.kafka.common.Uuid;
+
+import java.util.Objects;
+
+public class TopicIdPartition {
+ private final Uuid topicId;
+ private final Integer partition;
+
+ public TopicIdPartition(Uuid topicId, Integer topicPartition) {
+ this.topicId = Objects.requireNonNull(topicId, "topicId can not be null");
+ this.partition = Objects.requireNonNull(topicPartition, "topicPartition can not be null");
+ }
+
+ /**
+ * @return Universally unique id representing this topic partition.
+ */
+ public Uuid topicId() {
+ return topicId;
+ }
+
+ /**
+ * @return the partition id.
+ */
+ public int partition() {
+ return partition;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ TopicIdPartition that = (TopicIdPartition) o;
+ return topicId.equals(that.topicId) &&
+ partition.equals(that.partition);
+ }
+
+ @Override
+ public int hashCode() {
+ final int prime = 31;
+ int result = prime + topicId.hashCode();
+ result = prime * result + partition.hashCode();
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "TopicIdPartition(" +
+ "topicId=" + topicId +
+ ", partition=" + partition +
+ ')';
+ }
+}
Review Comment:
newline
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentAssignment The partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment,
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ Iterator<TopicIdPartition> partitionIterator = unassignedPartitions.iterator();
+ while (partitionIterator.hasNext()) {
+ TopicIdPartition partition = partitionIterator.next();
+ boolean assigned = false;
+
+ if (rackInfo.useRackStrategy && currentPartitionOwners.containsKey(partition)) {
+ String prevOwner = currentPartitionOwners.get(partition);
+ if (potentiallyUnfilledMembers.containsKey(prevOwner)) {
+ assigned = maybeAssignPartitionToMember(prevOwner, partition);
+ if (!potentiallyUnfilledMembers.containsKey(prevOwner)) {
+ roundRobinMembers.remove(prevOwner);
+ }
+ }
+ }
+
+ for (int i = 0; i < roundRobinMembers.size() && !assigned; i++) {
+ String memberId = roundRobinMembers.poll();
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ assigned = maybeAssignPartitionToMember(memberId, partition);
+ }
+ // Only re-add the member to the end of the queue if it's still available for assignment.
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ roundRobinMembers.add(memberId);
+ }
+ }
+
+ if (assigned) {
+ partitionIterator.remove();
+ }
+ }
+ }
+
+ /**
+ * Assigns the specified partition to the given member and updates the potentially unfilled members map.
+ * Only assign extra partitions once the member has met its minimum quota = total partitions / total members.
+ *
+ * <ol>
+ * <li> If the minimum quota hasn't been met aka remaining > 0 directly assign the partition.
+ * After assigning the partition, if the min quota has been met aka remaining = 0, remove the member
+ * if there's no members left to receive an extra partition. Otherwise, keep it in the
+ * potentially unfilled map. </li>
+ * <li> If the minimum quota has been met and if there is potential to receive an extra partition, assign it.
+ * Remove the member from the potentially unfilled map since it has already received the extra partition
+ * and met the min quota. </li>
+ * <li> Else, don't assign the partition. </li>
+ * </ol>
+ *
+ * @param memberId The Id of the member to which the partition will be assigned.
+ * @param topicIdPartition The topicIdPartition to be assigned.
+ */
+ private boolean maybeAssignPartitionToMember(String memberId, TopicIdPartition topicIdPartition) {
+ int remaining = potentiallyUnfilledMembers.get(memberId);
+ boolean shouldAssign = false;
+
+ // If the member hasn't met the minimum quota, set the flag for assignment.
+ // If member has met minimum quota and there's an extra partition available, set the flag for assignment.
+ if (remaining > 0) {
+ potentiallyUnfilledMembers.put(memberId, --remaining);
+ shouldAssign = true;
+
+ // If the member meets the minimum quota due to this assignment,
+ // check if any extra partitions are available.
+ // Removing the member from the list reduces an iteration for when remaining = 0 but there's no extras left.
+ if (remaining == 0 && remainingMembersToGetAnExtraPartition == 0) {
+ potentiallyUnfilledMembers.remove(memberId);
+ }
+ } else if (remaining == 0 && remainingMembersToGetAnExtraPartition > 0) {
+ remainingMembersToGetAnExtraPartition--;
+ // Each member can only receive one extra partition, once they meet the minimum quota and receive an extra
+ // partition they can be removed from the potentially unfilled members map.
+ potentiallyUnfilledMembers.remove(memberId);
+ shouldAssign = true;
+ }
+
+ // Assign the partition if flag is set.
+ if (shouldAssign) {
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ return true;
+ }
+
+ // No assignment possible because the
+ // Member met the minimum quota but number of members to receive an extra partition is zero.
+ return false;
+ }
+}
Review Comment:
newline
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentAssignment The partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment,
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ Iterator<TopicIdPartition> partitionIterator = unassignedPartitions.iterator();
+ while (partitionIterator.hasNext()) {
+ TopicIdPartition partition = partitionIterator.next();
+ boolean assigned = false;
+
+ if (rackInfo.useRackStrategy && currentPartitionOwners.containsKey(partition)) {
+ String prevOwner = currentPartitionOwners.get(partition);
+ if (potentiallyUnfilledMembers.containsKey(prevOwner)) {
+ assigned = maybeAssignPartitionToMember(prevOwner, partition);
+ if (!potentiallyUnfilledMembers.containsKey(prevOwner)) {
+ roundRobinMembers.remove(prevOwner);
+ }
+ }
+ }
+
+ for (int i = 0; i < roundRobinMembers.size() && !assigned; i++) {
+ String memberId = roundRobinMembers.poll();
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ assigned = maybeAssignPartitionToMember(memberId, partition);
+ }
+ // Only re-add the member to the end of the queue if it's still available for assignment.
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ roundRobinMembers.add(memberId);
+ }
+ }
+
+ if (assigned) {
+ partitionIterator.remove();
+ }
+ }
+ }
+
+ /**
+ * Assigns the specified partition to the given member and updates the potentially unfilled members map.
+ * Only assign extra partitions once the member has met its minimum quota = total partitions / total members.
+ *
+ * <ol>
+ * <li> If the minimum quota hasn't been met aka remaining > 0 directly assign the partition.
+ * After assigning the partition, if the min quota has been met aka remaining = 0, remove the member
+ * if there's no members left to receive an extra partition. Otherwise, keep it in the
+ * potentially unfilled map. </li>
+ * <li> If the minimum quota has been met and if there is potential to receive an extra partition, assign it.
+ * Remove the member from the potentially unfilled map since it has already received the extra partition
+ * and met the min quota. </li>
+ * <li> Else, don't assign the partition. </li>
+ * </ol>
+ *
+ * @param memberId The Id of the member to which the partition will be assigned.
+ * @param topicIdPartition The topicIdPartition to be assigned.
+ */
+ private boolean maybeAssignPartitionToMember(String memberId, TopicIdPartition topicIdPartition) {
+ int remaining = potentiallyUnfilledMembers.get(memberId);
+ boolean shouldAssign = false;
+
+ // If the member hasn't met the minimum quota, set the flag for assignment.
+ // If member has met minimum quota and there's an extra partition available, set the flag for assignment.
+ if (remaining > 0) {
+ potentiallyUnfilledMembers.put(memberId, --remaining);
+ shouldAssign = true;
+
+ // If the member meets the minimum quota due to this assignment,
+ // check if any extra partitions are available.
+ // Removing the member from the list reduces an iteration for when remaining = 0 but there's no extras left.
+ if (remaining == 0 && remainingMembersToGetAnExtraPartition == 0) {
+ potentiallyUnfilledMembers.remove(memberId);
+ }
+ } else if (remaining == 0 && remainingMembersToGetAnExtraPartition > 0) {
+ remainingMembersToGetAnExtraPartition--;
+ // Each member can only receive one extra partition, once they meet the minimum quota and receive an extra
+ // partition they can be removed from the potentially unfilled members map.
+ potentiallyUnfilledMembers.remove(memberId);
+ shouldAssign = true;
+ }
+
+ // Assign the partition if flag is set.
+ if (shouldAssign) {
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ return true;
+ }
+
+ // No assignment possible because the
+ // Member met the minimum quota but number of members to receive an extra partition is zero.
Review Comment:
can we reformat this?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentAssignment The partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment,
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ Iterator<TopicIdPartition> partitionIterator = unassignedPartitions.iterator();
+ while (partitionIterator.hasNext()) {
+ TopicIdPartition partition = partitionIterator.next();
+ boolean assigned = false;
+
+ if (rackInfo.useRackStrategy && currentPartitionOwners.containsKey(partition)) {
+ String prevOwner = currentPartitionOwners.get(partition);
+ if (potentiallyUnfilledMembers.containsKey(prevOwner)) {
+ assigned = maybeAssignPartitionToMember(prevOwner, partition);
+ if (!potentiallyUnfilledMembers.containsKey(prevOwner)) {
+ roundRobinMembers.remove(prevOwner);
+ }
+ }
+ }
+
+ for (int i = 0; i < roundRobinMembers.size() && !assigned; i++) {
+ String memberId = roundRobinMembers.poll();
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ assigned = maybeAssignPartitionToMember(memberId, partition);
+ }
+ // Only re-add the member to the end of the queue if it's still available for assignment.
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ roundRobinMembers.add(memberId);
+ }
+ }
+
+ if (assigned) {
+ partitionIterator.remove();
+ }
+ }
+ }
+
+ /**
+ * Assigns the specified partition to the given member and updates the potentially unfilled members map.
+ * Only assign extra partitions once the member has met its minimum quota = total partitions / total members.
+ *
+ * <ol>
+ * <li> If the minimum quota hasn't been met aka remaining > 0 directly assign the partition.
+ * After assigning the partition, if the min quota has been met aka remaining = 0, remove the member
+ * if there's no members left to receive an extra partition. Otherwise, keep it in the
+ * potentially unfilled map. </li>
+ * <li> If the minimum quota has been met and if there is potential to receive an extra partition, assign it.
+ * Remove the member from the potentially unfilled map since it has already received the extra partition
+ * and met the min quota. </li>
+ * <li> Else, don't assign the partition. </li>
+ * </ol>
+ *
+ * @param memberId The Id of the member to which the partition will be assigned.
+ * @param topicIdPartition The topicIdPartition to be assigned.
+ */
Review Comment:
let's add @return docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1378163030
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> When rack aware strategy is used, only partitions with their rack matching their current
+ * owner's rack are returned in the valid assignment.</li>
+ * <li> If the current assignment exists, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentMemberAssignment = validCurrentMemberAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentMemberAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member based on certain criteria.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentMemberAssignment The map of topics to partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentMemberAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentMemberAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentMemberAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ List<TopicIdPartition> sortedPartitionsList = unassignedPartitions.stream()
+ .sorted(Comparator.comparing(TopicIdPartition::topicId).thenComparing(TopicIdPartition::partitionId))
+ .collect(Collectors.toList());
Review Comment:
Sorted so that partitions from different topics are distributed equally amongst all the consumers, as a bonus it's easier to predict which partitions go to which consumer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340487558
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
Review Comment:
yes, the javadoc for validCurrentassignment mentions this, but I added another line just for clarity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1380818617
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a set of {@code TopicIdPartition} including all the given topic Ids based on their partition counts.
+ *
+ * @param topicIds The topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return Set of {@code TopicIdPartition} including all the provided topic Ids.
+ */
+ protected static Set<TopicIdPartition> topicIdPartitions(
+ Collection<Uuid> topicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ return topicIds.stream()
+ .flatMap(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .mapToObj(i -> new TopicIdPartition(topic, i))
+ ).collect(Collectors.toSet());
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * List of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> racksPerPartition;
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ racksPerPartition = Collections.emptyMap();
+ } else {
+ racksPerPartition = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions(topicIds, subscribedTopicDescriber).forEach(tp -> {
Review Comment:
okie I get it now, I added a lambda function! thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1380777612
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscribedTopicIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscribedTopicIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscribedTopicIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscribedTopicIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscribedTopicIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
Review Comment:
yep sounds good, I'll move it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1380756376
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> When rack aware strategy is used, only partitions with their rack matching their current
+ * owner's rack are returned in the valid assignment.</li>
+ * <li> If the current assignment exists, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentMemberAssignment = validCurrentMemberAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentMemberAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member based on certain criteria.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentMemberAssignment The map of topics to partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentMemberAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentMemberAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentMemberAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ List<TopicIdPartition> sortedPartitionsList = unassignedPartitions.stream()
+ .sorted(Comparator.comparing(TopicIdPartition::topicId).thenComparing(TopicIdPartition::partitionId))
+ .collect(Collectors.toList());
Review Comment:
yep sure will add it! I'll probably forget too lol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1377670781
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
Review Comment:
nit: Let's remove an empty line here.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
Review Comment:
nit: `subscribedTopicIds`?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * List of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> racksPerPartition;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ racksPerPartition = Collections.emptyMap();
+ } else {
+ racksPerPartition = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions.forEach(tp -> {
Review Comment:
It is actually a bit wasteful to generate a List to just iterate once on it, no?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * List of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> racksPerPartition;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
Review Comment:
It seems that we only need this in the else branch below. Should we move it there?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
Review Comment:
In `allTopicIdPartitions`, we sort the list and then put it in a HashSet. Could we optimize this?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> When rack aware strategy is used, only partitions with their rack matching their current
+ * owner's rack are returned in the valid assignment.</li>
+ * <li> If the current assignment exists, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentMemberAssignment = validCurrentMemberAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentMemberAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member based on certain criteria.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentMemberAssignment The map of topics to partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentMemberAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentMemberAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentMemberAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ List<TopicIdPartition> sortedPartitionsList = unassignedPartitions.stream()
+ .sorted(Comparator.comparing(TopicIdPartition::topicId).thenComparing(TopicIdPartition::partitionId))
+ .collect(Collectors.toList());
Review Comment:
I don't recall. Why do we need to sort the partitions here?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadata.java:
##########
@@ -68,12 +68,12 @@ public int numPartitions(Uuid topicId) {
* @param topicId Uuid corresponding to the partition's topic.
* @param partition Partition Id within the topic.
* @return The set of racks corresponding to the replicas of the topics partition.
- * If the topic Id does not exist, an empty set is returned
+ * If the topic Id does not exist or no partition rack information is available, an empty set is returned.
*/
@Override
public Set<String> racksForPartition(Uuid topicId, int partition) {
TopicMetadata topic = this.topicMetadata.get(topicId);
- return topic == null ? Collections.emptySet() : topic.partitionRacks().get(partition);
+ return topic == null ? Collections.emptySet() : topic.partitionRacks().getOrDefault(partition, Collections.emptySet());
Review Comment:
Should we add a test to SubscribedTopicMetadataTest for this one?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> When rack aware strategy is used, only partitions with their rack matching their current
+ * owner's rack are returned in the valid assignment.</li>
+ * <li> If the current assignment exists, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentMemberAssignment = validCurrentMemberAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentMemberAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member based on certain criteria.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentMemberAssignment The map of topics to partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentMemberAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentMemberAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentMemberAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ List<TopicIdPartition> sortedPartitionsList = unassignedPartitions.stream()
+ .sorted(Comparator.comparing(TopicIdPartition::topicId).thenComparing(TopicIdPartition::partitionId))
+ .collect(Collectors.toList());
+
+ Iterator<TopicIdPartition> partitionIterator = sortedPartitionsList.iterator();
+ while (partitionIterator.hasNext()) {
+ TopicIdPartition topicIdPartition = partitionIterator.next();
+ boolean assigned = false;
+
+ if (rackInfo.useRackStrategy && currentPartitionOwners.containsKey(topicIdPartition)) {
+ String currentOwner = currentPartitionOwners.get(topicIdPartition);
+ if (potentiallyUnfilledMembers.containsKey(currentOwner)) {
+ assigned = maybeAssignPartitionToMember(currentOwner, topicIdPartition);
+ if (!potentiallyUnfilledMembers.containsKey(currentOwner)) {
+ roundRobinMembers.remove(currentOwner);
+ }
+ }
+ }
+
+ for (int i = 0; i < roundRobinMembers.size() && !assigned; i++) {
+ String memberId = roundRobinMembers.poll();
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ assigned = maybeAssignPartitionToMember(memberId, topicIdPartition);
+ }
+ // Only re-add the member to the end of the queue if it's still available for assignment.
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ roundRobinMembers.add(memberId);
+ }
+ }
+
+ if (assigned) {
+ partitionIterator.remove();
Review Comment:
Why do we need to remove the partition from the `sortedPartitionsList`? It does not seem necessary to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1377864177
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
Review Comment:
Yep, also on second look I also realized I don't use the sorted list from allTopicIdPartitions anymore since I changed the implementation so I can just remove the sorting altogether. Missed this side effect from the changes, thanks for the catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1378163455
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * List of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> racksPerPartition;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ racksPerPartition = Collections.emptyMap();
+ } else {
+ racksPerPartition = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions.forEach(tp -> {
Review Comment:
yes agreed, I changed it now, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1380551541
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a set of {@code TopicIdPartition} including all the given topic Ids based on their partition counts.
+ *
+ * @param topicIds The topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return Set of {@code TopicIdPartition} including all the provided topic Ids.
+ */
+ protected static Set<TopicIdPartition> topicIdPartitions(
+ Collection<Uuid> topicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ return topicIds.stream()
+ .flatMap(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .mapToObj(i -> new TopicIdPartition(topic, i))
+ ).collect(Collectors.toSet());
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * List of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> racksPerPartition;
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ racksPerPartition = Collections.emptyMap();
+ } else {
+ racksPerPartition = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions(topicIds, subscribedTopicDescriber).forEach(tp -> {
Review Comment:
I think that my previous comment still holds here. It is a bit wasteful to generate a list with topicIdPartitions to just iterate over it. I wonder if we could either directly have a loop here or have an helper method which takes a lambda to do so.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadataTest.java:
##########
@@ -43,27 +45,62 @@ public void testAttribute() {
new TopicMetadata(topicId, topicName, 5, partitionRacks)
);
}
- assertEquals(topicMetadataMap, new SubscribedTopicMetadata(topicMetadataMap).topicMetadata());
+ subscribedTopicMetadata = new SubscribedTopicMetadata(topicMetadataMap);
+ }
+
+
+ @Test
+ public void testAttribute() {
+ setup();
Review Comment:
Instead of doing this, we could use the `@BeforeEach` annotation. You will find examples in the code base.
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadataTest.java:
##########
@@ -43,27 +45,62 @@ public void testAttribute() {
new TopicMetadata(topicId, topicName, 5, partitionRacks)
);
}
- assertEquals(topicMetadataMap, new SubscribedTopicMetadata(topicMetadataMap).topicMetadata());
+ subscribedTopicMetadata = new SubscribedTopicMetadata(topicMetadataMap);
+ }
+
+
+ @Test
+ public void testAttribute() {
+ setup();
+ assertEquals(topicMetadataMap, subscribedTopicMetadata.topicMetadata());
}
@Test
public void testTopicMetadataCannotBeNull() {
assertThrows(NullPointerException.class, () -> new SubscribedTopicMetadata(null));
}
+ @Test
Review Comment:
nit: Empty line before.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a set of {@code TopicIdPartition} including all the given topic Ids based on their partition counts.
+ *
+ * @param topicIds The topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return Set of {@code TopicIdPartition} including all the provided topic Ids.
+ */
+ protected static Set<TopicIdPartition> topicIdPartitions(
+ Collection<Uuid> topicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ return topicIds.stream()
+ .flatMap(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .mapToObj(i -> new TopicIdPartition(topic, i))
Review Comment:
nit: The `mapToObj` seems misplaced. How about?
```
IntStream
.range(0, subscribedTopicDescriber.numPartitions(topic))
.mapToObj(i -> new TopicIdPartition(topic, i))
```
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadataTest.java:
##########
@@ -43,27 +45,62 @@ public void testAttribute() {
new TopicMetadata(topicId, topicName, 5, partitionRacks)
);
}
- assertEquals(topicMetadataMap, new SubscribedTopicMetadata(topicMetadataMap).topicMetadata());
+ subscribedTopicMetadata = new SubscribedTopicMetadata(topicMetadataMap);
+ }
+
+
+ @Test
+ public void testAttribute() {
+ setup();
+ assertEquals(topicMetadataMap, subscribedTopicMetadata.topicMetadata());
}
@Test
public void testTopicMetadataCannotBeNull() {
assertThrows(NullPointerException.class, () -> new SubscribedTopicMetadata(null));
}
+ @Test
+ public void testNumberOfPartitions() {
+ setup();
+ Uuid topicId = Uuid.randomUuid();
+
+ // Test -1 is returned when the topic Id doesn't exist.
+ assertEquals(-1, subscribedTopicMetadata.numPartitions(topicId));
+
+ topicMetadataMap.put(topicId, new TopicMetadata(topicId, "topic6", 3, Collections.emptyMap()));
+
+ // Test that the correct number of partitions are returned for a given topic Id.
+ assertEquals(3, subscribedTopicMetadata.numPartitions(topicId));
+ }
+ @Test
Review Comment:
nit: Empty line before.
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscribedTopicIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscribedTopicIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscribedTopicIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscribedTopicIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscribedTopicIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
Review Comment:
nit: Should we put this block as the first thing in this method?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> When rack aware strategy is used, only partitions with their rack matching their current
+ * owner's rack are returned in the valid assignment.</li>
+ * <li> If the current assignment exists, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentMemberAssignment = validCurrentMemberAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentMemberAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member based on certain criteria.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentMemberAssignment The map of topics to partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentMemberAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentMemberAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentMemberAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ List<TopicIdPartition> sortedPartitionsList = unassignedPartitions.stream()
+ .sorted(Comparator.comparing(TopicIdPartition::topicId).thenComparing(TopicIdPartition::partitionId))
+ .collect(Collectors.toList());
Review Comment:
Understood. Should we add a small comment about it? I will very likely forget again... ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1380781750
##########
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadataTest.java:
##########
@@ -43,27 +45,62 @@ public void testAttribute() {
new TopicMetadata(topicId, topicName, 5, partitionRacks)
);
}
- assertEquals(topicMetadataMap, new SubscribedTopicMetadata(topicMetadataMap).topicMetadata());
+ subscribedTopicMetadata = new SubscribedTopicMetadata(topicMetadataMap);
+ }
+
+
+ @Test
+ public void testAttribute() {
+ setup();
Review Comment:
thankss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1377844645
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,419 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private Set<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = new HashSet<>(allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber));
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> When rack aware strategy is used, only partitions with their rack matching their current
+ * owner's rack are returned in the valid assignment.</li>
+ * <li> If the current assignment exists, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentMemberAssignment = validCurrentMemberAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentMemberAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ });
+
+ // The extra partition is located at the last index from the previous step.
+ if (remaining < 0 && remainingMembersToGetAnExtraPartition > 0) {
+ TopicIdPartition topicIdPartition = validCurrentMemberAssignment.get(retainedPartitionsCount);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partitionId()
+ );
+ unassignedPartitions.remove(topicIdPartition);
+ remainingMembersToGetAnExtraPartition--;
+ }
+ }
+
+ if (remaining >= 0) {
+ potentiallyUnfilledMembers.put(memberId, remaining);
+ }
+ });
+
+ return potentiallyUnfilledMembers;
+ }
+
+ /**
+ * Filters the current assignment of partitions for a given member based on certain criteria.
+ *
+ * Any partition that still belongs to the member's subscribed topics list is considered valid.
+ * If rack aware strategy can be used: Only partitions with matching rack are valid and non-matching partitions are
+ * tracked with their current owner for future use.
+ *
+ * @param memberId The Id of the member whose assignment is being validated.
+ * @param currentMemberAssignment The map of topics to partitions currently assigned to the member.
+ *
+ * @return List of valid partitions after applying the filters.
+ */
+ private List<TopicIdPartition> validCurrentMemberAssignment(
+ String memberId,
+ Map<Uuid, Set<Integer>> currentMemberAssignment
+ ) {
+ List<TopicIdPartition> validCurrentAssignmentList = new ArrayList<>();
+ currentMemberAssignment.forEach((topicId, partitions) -> {
+ if (subscriptionIds.contains(topicId)) {
+ partitions.forEach(partition -> {
+ TopicIdPartition topicIdPartition = new TopicIdPartition(topicId, partition);
+ if (rackInfo.useRackStrategy && rackInfo.racksMismatch(memberId, topicIdPartition)) {
+ currentPartitionOwners.put(topicIdPartition, memberId);
+ } else {
+ validCurrentAssignmentList.add(topicIdPartition);
+ }
+ });
+ } else {
+ LOG.debug("The topic " + topicId + " is no longer present in the subscribed topics list");
+ }
+ });
+
+ return validCurrentAssignmentList;
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members present in the same rack.
+ * Partitions with the least number of potential members in the same rack are allotted first.
+ * Members in the same rack with the least number of partitions in the target assignment
+ * are assigned partitions first.
+ */
+ private void rackAwarePartitionAssignment() {
+ // Sort partitions in ascending order by number of potential members with matching racks.
+ // Partitions with no potential members in the same rack aren't included in this list.
+ List<TopicIdPartition> sortedPartitions = rackInfo.sortPartitionsByRackMembers(unassignedPartitions);
+
+ sortedPartitions.forEach(partition -> {
+ List<String> sortedMembersWithMatchingRack = rackInfo.getSortedMembersWithMatchingRack(partition, targetAssignment);
+
+ for (String memberId : sortedMembersWithMatchingRack) {
+ if (potentiallyUnfilledMembers.containsKey(memberId) && maybeAssignPartitionToMember(memberId, partition)) {
+ unassignedPartitions.remove(partition);
+ break;
+ }
+ }
+ });
+ }
+
+ /**
+ * Allocates the unassigned partitions to unfilled members in a round-robin fashion.
+ *
+ * If the rack-aware strategy is enabled, partitions are attempted to be assigned back to their current owners first.
+ * This is because pure stickiness without rack matching is not considered initially.
+ *
+ * If a partition couldn't be assigned to its current owner due to the quotas OR
+ * if the rack-aware strategy is not enabled, the partitions are allocated to the unfilled members.
+ */
+ private void unassignedPartitionsRoundRobinAssignment() {
+ Queue<String> roundRobinMembers = new LinkedList<>(potentiallyUnfilledMembers.keySet());
+
+ List<TopicIdPartition> sortedPartitionsList = unassignedPartitions.stream()
+ .sorted(Comparator.comparing(TopicIdPartition::topicId).thenComparing(TopicIdPartition::partitionId))
+ .collect(Collectors.toList());
+
+ Iterator<TopicIdPartition> partitionIterator = sortedPartitionsList.iterator();
+ while (partitionIterator.hasNext()) {
+ TopicIdPartition topicIdPartition = partitionIterator.next();
+ boolean assigned = false;
+
+ if (rackInfo.useRackStrategy && currentPartitionOwners.containsKey(topicIdPartition)) {
+ String currentOwner = currentPartitionOwners.get(topicIdPartition);
+ if (potentiallyUnfilledMembers.containsKey(currentOwner)) {
+ assigned = maybeAssignPartitionToMember(currentOwner, topicIdPartition);
+ if (!potentiallyUnfilledMembers.containsKey(currentOwner)) {
+ roundRobinMembers.remove(currentOwner);
+ }
+ }
+ }
+
+ for (int i = 0; i < roundRobinMembers.size() && !assigned; i++) {
+ String memberId = roundRobinMembers.poll();
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ assigned = maybeAssignPartitionToMember(memberId, topicIdPartition);
+ }
+ // Only re-add the member to the end of the queue if it's still available for assignment.
+ if (potentiallyUnfilledMembers.containsKey(memberId)) {
+ roundRobinMembers.add(memberId);
+ }
+ }
+
+ if (assigned) {
+ partitionIterator.remove();
Review Comment:
you're right, thanks for pointing it out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1378192398
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/SubscribedTopicMetadata.java:
##########
@@ -68,12 +68,12 @@ public int numPartitions(Uuid topicId) {
* @param topicId Uuid corresponding to the partition's topic.
* @param partition Partition Id within the topic.
* @return The set of racks corresponding to the replicas of the topics partition.
- * If the topic Id does not exist, an empty set is returned
+ * If the topic Id does not exist or no partition rack information is available, an empty set is returned.
*/
@Override
public Set<String> racksForPartition(Uuid topicId, int partition) {
TopicMetadata topic = this.topicMetadata.get(topicId);
- return topic == null ? Collections.emptySet() : topic.partitionRacks().get(partition);
+ return topic == null ? Collections.emptySet() : topic.partitionRacks().getOrDefault(partition, Collections.emptySet());
Review Comment:
good idea, I'll add for numPartitions as well, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340497083
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
Review Comment:
So from what I uderstood I made validCurrentMemberAssignment and changed the args name too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA 14515: Optimized Uniform Rack Aware Assignor [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1380768591
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.server.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param allPartitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> allPartitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, allPartitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(allPartitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignments,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignments.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a set of {@code TopicIdPartition} including all the given topic Ids based on their partition counts.
+ *
+ * @param topicIds The topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return Set of {@code TopicIdPartition} including all the provided topic Ids.
+ */
+ protected static Set<TopicIdPartition> topicIdPartitions(
+ Collection<Uuid> topicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ return topicIds.stream()
+ .flatMap(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .mapToObj(i -> new TopicIdPartition(topic, i))
Review Comment:
oh yep changed it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340481221
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AbstractUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * The assignment builder is used to construct the target assignment based on the members' subscriptions.
+ *
+ * This class contains common utility methods and a class for obtaining and storing rack information.
+ */
+public abstract class AbstractUniformAssignmentBuilder {
+ protected abstract GroupAssignment buildAssignment();
+
+ /**
+ * Determines if rack-aware assignment is appropriate based on the provided rack information.
+ *
+ * @param memberRacks Racks where members are located.
+ * @param partitionRacks Racks where partitions are located.
+ * @param racksPerPartition Map of partitions to their associated racks.
+ *
+ * @return {@code true} if rack-aware assignment should be applied; {@code false} otherwise.
+ */
+ protected static boolean useRackAwareAssignment(
+ Set<String> memberRacks,
+ Set<String> partitionRacks,
+ Map<TopicIdPartition, Set<String>> racksPerPartition
+ ) {
+ if (memberRacks.isEmpty() || Collections.disjoint(memberRacks, partitionRacks))
+ return false;
+ else {
+ return !racksPerPartition.values().stream().allMatch(partitionRacks::equals);
+ }
+ }
+
+ /**
+ * Adds the topic's partition to the member's target assignment.
+ */
+ protected static void addPartitionToAssignment(
+ Map<String, MemberAssignment> memberAssignment,
+ String memberId,
+ Uuid topicId,
+ int partition
+ ) {
+ memberAssignment.get(memberId)
+ .targetPartitions()
+ .computeIfAbsent(topicId, __ -> new HashSet<>())
+ .add(partition);
+ }
+
+ /**
+ * Constructs a list of {@code TopicIdPartition} for each topic Id based on its partition count.
+ *
+ * @param allTopicIds The subscribed topic Ids.
+ * @param subscribedTopicDescriber Utility to fetch the partition count for a given topic.
+ *
+ * @return List of sorted {@code TopicIdPartition} for all provided topic Ids.
+ */
+ protected static List<TopicIdPartition> allTopicIdPartitions(
+ Collection<Uuid> allTopicIds,
+ SubscribedTopicDescriber subscribedTopicDescriber
+ ) {
+ List<TopicIdPartition> allTopicIdPartitions = new ArrayList<>();
+ // Sorted so that partitions from each topic can be distributed amongst its subscribers equally.
+ allTopicIds.stream().sorted().forEach(topic ->
+ IntStream.range(0, subscribedTopicDescriber.numPartitions(topic))
+ .forEach(i -> allTopicIdPartitions.add(new TopicIdPartition(topic, i)))
+ );
+
+ return allTopicIdPartitions;
+ }
+
+ /**
+ * Represents the rack information of members and partitions along with utility methods
+ * to facilitate rack-aware assignment strategies for a given consumer group.
+ */
+ protected static class RackInfo {
+ /**
+ * Map of every member to its rack.
+ */
+ protected final Map<String, String> memberRacks;
+
+ /**
+ * Map of every partition to a list of its racks.
+ */
+ protected final Map<TopicIdPartition, Set<String>> partitionRacks;
+
+ /**
+ * Number of members with the same rack as the partition.
+ */
+ protected final Map<TopicIdPartition, List<String>> membersWithSameRackAsPartition;
+
+ /**
+ * Indicates if a rack aware assignment can be done.
+ * True if racks are defined for both members and partitions and there is an intersection between the sets.
+ */
+ protected final boolean useRackStrategy;
+
+ /**
+ * Constructs rack information based on the assignment specification and subscribed topics.
+ *
+ * @param assignmentSpec The current assignment specification.
+ * @param subscribedTopicDescriber Topic and partition metadata of the subscribed topics.
+ * @param topicIds List of topic Ids.
+ */
+ public RackInfo(
+ AssignmentSpec assignmentSpec,
+ SubscribedTopicDescriber subscribedTopicDescriber,
+ Set<Uuid> topicIds
+ ) {
+ Map<String, List<String>> membersByRack = new HashMap<>();
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) ->
+ assignmentMemberSpec.rackId().filter(r -> !r.isEmpty()).ifPresent(
+ rackId -> membersByRack.computeIfAbsent(rackId, __ -> new ArrayList<>()).add(memberId)
+ )
+ );
+
+ Set<String> allPartitionRacks;
+ Map<TopicIdPartition, Set<String>> partitionRacks;
+ List<TopicIdPartition> topicIdPartitions = allTopicIdPartitions(topicIds, subscribedTopicDescriber);
+
+ if (membersByRack.isEmpty()) {
+ allPartitionRacks = Collections.emptySet();
+ partitionRacks = Collections.emptyMap();
+ } else {
+ partitionRacks = new HashMap<>();
+ allPartitionRacks = new HashSet<>();
+ topicIdPartitions.forEach(tp -> {
+ Set<String> racks = subscribedTopicDescriber.racksForPartition(tp.topicId(), tp.partition());
+ partitionRacks.put(tp, racks);
+ if (!racks.isEmpty()) allPartitionRacks.addAll(racks);
+ });
+ }
+
+ if (useRackAwareAssignment(membersByRack.keySet(), allPartitionRacks, partitionRacks)) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340483136
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GeneralUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(GeneralUniformAssignmentBuilder.class);
+
+ @Override
+ protected GroupAssignment buildAssignment() {
+ return null;
+ }
+}
Review Comment:
ignore this file cause the next PR will take care of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rreddy-22 commented on a diff in pull request #14416: KAFKA 14515: Optimized Uniform Rack Aware Assignor
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #14416:
URL: https://github.com/apache/kafka/pull/14416#discussion_r1340500692
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/OptimizedUniformAssignmentBuilder.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.coordinator.group.assignor;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.coordinator.group.common.TopicIdPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.IntStream;
+
+import static java.lang.Math.min;
+
+/**
+ * The optimized uniform assignment builder is used to generate the target assignment for a consumer group with
+ * all its members subscribed to the same set of topics.
+ * It is optimized since the assignment can be done in fewer, less complicated steps compared to when
+ * the subscriptions are different across the members.
+ *
+ * Assignments are done according to the following principles:
+ *
+ *
+ * <li> Balance: Ensure partitions are distributed equally among all members.
+ * The difference in assignments sizes between any two members
+ * should not exceed one partition. </li>
+ * <li> Rack Matching: When feasible, aim to assign partitions to members
+ * located on the same rack thus avoiding cross-zone traffic. </li>
+ * <li> Stickiness: Minimize partition movements among members by retaining
+ * as much of the existing assignment as possible. </li>
+ *
+ * The assignment builder prioritizes the properties in the following order:
+ * Balance > Rack Matching > Stickiness.
+ */
+public class OptimizedUniformAssignmentBuilder extends AbstractUniformAssignmentBuilder {
+ private static final Logger LOG = LoggerFactory.getLogger(OptimizedUniformAssignmentBuilder.class);
+
+ /**
+ * The assignment specification which includes member metadata.
+ */
+ private final AssignmentSpec assignmentSpec;
+
+ /**
+ * The topic and partition metadata describer.
+ */
+ private final SubscribedTopicDescriber subscribedTopicDescriber;
+
+ /**
+ * The set of topic Ids that the consumer group is subscribed to.
+ */
+ private final Set<Uuid> subscriptionIds;
+
+ /**
+ * Rack information and helper methods.
+ */
+ private final RackInfo rackInfo;
+
+ /**
+ * The number of members to receive an extra partition beyond the minimum quota.
+ * Minimum Quota = Total Partitions / Total Members
+ * Example: If there are 11 partitions to be distributed among 3 members,
+ * each member gets 3 (11 / 3) [minQuota] partitions and 2 (11 % 3) members get an extra partition.
+ */
+ private int remainingMembersToGetAnExtraPartition;
+
+ /**
+ * Members mapped to the remaining number of partitions needed to meet the minimum quota.
+ * Minimum quota = total partitions / total members.
+ */
+ private Map<String, Integer> potentiallyUnfilledMembers;
+
+ /**
+ * The partitions that still need to be assigned.
+ * Initially this contains all the subscribed topics' partitions.
+ */
+ private List<TopicIdPartition> unassignedPartitions;
+
+ /**
+ * The target assignment.
+ */
+ private final Map<String, MemberAssignment> targetAssignment;
+
+ /**
+ * Tracks the existing owner of each partition.
+ * Only populated when the rack awareness strategy is used.
+ */
+ private final Map<TopicIdPartition, String> currentPartitionOwners;
+
+ OptimizedUniformAssignmentBuilder(AssignmentSpec assignmentSpec, SubscribedTopicDescriber subscribedTopicDescriber) {
+ this.assignmentSpec = assignmentSpec;
+ this.subscribedTopicDescriber = subscribedTopicDescriber;
+ this.subscriptionIds = new HashSet<>(assignmentSpec.members().values().iterator().next().subscribedTopicIds());
+ this.rackInfo = new RackInfo(assignmentSpec, subscribedTopicDescriber, subscriptionIds);
+ this.potentiallyUnfilledMembers = new HashMap<>();
+ this.targetAssignment = new HashMap<>();
+ // Without rack-aware strategy, tracking current owners of unassigned partitions is unnecessary
+ // as all sticky partitions are retained until a member meets its quota.
+ this.currentPartitionOwners = rackInfo.useRackStrategy ? new HashMap<>() : Collections.emptyMap();
+ }
+
+ /**
+ * Here's the step-by-step breakdown of the assignment process:
+ *
+ * <li> Compute the quotas of partitions for each member based on the total partitions and member count.</li>
+ * <li> Initialize unassigned partitions to all the topic partitions and
+ * remove partitions from the list as and when they are assigned.</li>
+ * <li> For existing assignments, retain partitions based on the determined quota and member's rack compatibility.</li>
+ * <li> If a partition's rack mismatches with its owner, track it for future use.</li>
+ * <li> Identify members that haven't fulfilled their partition quota or are eligible to receive extra partitions.</li>
+ * <li> Proceed with a round-robin assignment adhering to rack awareness.
+ * For each unassigned partition, locate the first compatible member from the potentially unfilled list.</li>
+ * <li> If no rack-compatible member is found, revert to the tracked current owner.
+ * If that member can't accommodate the partition due to quota limits, resort to a generic round-robin assignment.</li>
+ */
+ @Override
+ protected GroupAssignment buildAssignment() throws PartitionAssignorException {
+ int totalPartitionsCount = 0;
+
+ for (Uuid topicId : subscriptionIds) {
+ int partitionCount = subscribedTopicDescriber.numPartitions(topicId);
+ if (partitionCount == -1) {
+ throw new PartitionAssignorException(
+ "Members are subscribed to topic " + topicId + " which doesn't exist in the topic metadata."
+ );
+ } else {
+ totalPartitionsCount += partitionCount;
+ }
+ }
+
+ if (subscriptionIds.isEmpty()) {
+ LOG.debug("The subscription list is empty, returning an empty assignment");
+ return new GroupAssignment(Collections.emptyMap());
+ }
+
+ // The minimum required quota that each member needs to meet for a balanced assignment.
+ // This is the same for all members.
+ final int numberOfMembers = assignmentSpec.members().size();
+ final int minQuota = totalPartitionsCount / numberOfMembers;
+ remainingMembersToGetAnExtraPartition = totalPartitionsCount % numberOfMembers;
+
+ assignmentSpec.members().keySet().forEach(memberId ->
+ targetAssignment.put(memberId, new MemberAssignment(new HashMap<>())
+ ));
+
+ // Sorted list of all partitions.
+ unassignedPartitions = allTopicIdPartitions(subscriptionIds, subscribedTopicDescriber);
+ potentiallyUnfilledMembers = assignStickyPartitions(minQuota);
+
+ if (rackInfo.useRackStrategy) rackAwarePartitionAssignment();
+ unassignedPartitionsRoundRobinAssignment();
+
+ if (!unassignedPartitions.isEmpty()) {
+ throw new PartitionAssignorException("Partitions were left unassigned");
+ }
+
+ return new GroupAssignment(targetAssignment);
+ }
+
+ /**
+ * Retains a set of partitions from the existing assignment and includes them in the target assignment.
+ * Only relevant partitions that exist in the current topic metadata and subscriptions are considered.
+ * In addition, if rack awareness is enabled, it is ensured that a partition's rack matches the member's rack.
+ *
+ * <p>For each member:
+ * <ol>
+ * <li> Find the valid current assignment considering topic subscriptions, metadata and rack information.</li>
+ * <li> If the current assignments exist, retain partitions up to the minimum quota.</li>
+ * <li> If the current assignment size is greater than the minimum quota and
+ * there are members that could get an extra partition, assign the next partition as well.</li>
+ * <li> Finally, if the member's current assignment size is less than the minimum quota,
+ * add them to the potentially unfilled members map and track the number of remaining
+ * partitions required to meet the quota.</li>
+ * </ol>
+ * </p>
+ *
+ * @return Members mapped to the remaining number of partitions needed to meet the minimum quota,
+ * including members that are eligible to receive an extra partition.
+ */
+ private Map<String, Integer> assignStickyPartitions(int minQuota) {
+ Map<String, Integer> potentiallyUnfilledMembers = new HashMap<>();
+
+ assignmentSpec.members().forEach((memberId, assignmentMemberSpec) -> {
+ List<TopicIdPartition> validCurrentAssignment = validCurrentAssignment(
+ memberId,
+ assignmentMemberSpec.assignedPartitions()
+ );
+
+ int currentAssignmentSize = validCurrentAssignment.size();
+ // Number of partitions required to meet the minimum quota.
+ int remaining = minQuota - currentAssignmentSize;
+
+ if (currentAssignmentSize > 0) {
+ int retainedPartitionsCount = min(currentAssignmentSize, minQuota);
+ IntStream.range(0, retainedPartitionsCount).forEach(i -> {
+ TopicIdPartition topicIdPartition = validCurrentAssignment.get(i);
+ addPartitionToAssignment(
+ targetAssignment,
+ memberId,
+ topicIdPartition.topicId(),
+ topicIdPartition.partition()
+ );
+ unassignedPartitions.remove(topicIdPartition);
Review Comment:
We're not iterating through the whole list though? For each member you get a list of topicIdPartitions that are valid. Then based on the quotas we just pick the first few and add it to the target assignment and also remove it from unassigned partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org