You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "suzhiking (via GitHub)" <gi...@apache.org> on 2026/10/27 00:06:27 UTC

[PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

suzhiking opened a new pull request, #23969:
URL: https://github.com/apache/kafka/pull/23969

   `AssignmentRefinerImpl.refine()` runs on every streams group heartbeat. This PR makes a call cheaper without changing the intermediate assignment it derives. Each commit is one optimization and can be reviewed on its own:
   
   1. Return the target assignment right after the case analysis when it stages no migration.
   2. Walk every task fewer times: count process load in the index walk, find borrowed standby conflicts through the target standby holders, and build that lookup only when it is read.
   3. Sort only the decisions, not every stateful target active task.
   4. Skip counting active tasks when the refiner returns the target assignment itself.
   5. Resolve the stateful subtopologies once per call instead of per (member, subtopology) entry.
   6. Find the tasks not on their target owner first, and return before indexing when there are none. This assumes the target assignment names one member per active task, as the rest of the refiner already does.
   7. Compare task IDs directly instead of building a comparator chain per comparison.
   8. Index only the tasks not on their target owner; the process load still counts every task.
   9. Build the target standby holders only for those tasks.
   10. Patch members copy-on-write instead of copying every patched member in full.
   11. Keep borrowed and parked migrations in hash sets instead of tree sets.
   12. Count active tasks only for members whose tasks differ from the target assignment.
   13. Build the state-on-disk lookup only when the case analysis asks for it.
   
   Besides the unit tests, each step was checked against the previous refiner on 20,000 random groups, with identical results.
   
   `AssignmentRefinerBenchmark.refinerImpl`, 1,000 members, 5,000 stateful tasks, one standby, MIXED lag, budget 2, through step 10, µs per call:
   
   | Migrating | Before | After |
   |---|---|---|
   | 0% | 4,573 | 163 |
   | 1% | 4,760 | 1,240 |
   | 50% | 6,855 | 3,365 |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4620912472


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -158,6 +168,9 @@ static CurrentAssignmentIndex indexCurrentAssignment(
                 subtopologies,
                 task -> reportedByProcess.computeIfAbsent(member.processId(), __ -> new HashSet<>()).add(task)
             );
+
+            memberCounts.merge(member.processId(), 1, Integer::sum);
+            statefulTaskCounts.merge(member.processId(), statefulTaskCount[0], Integer::sum);

Review Comment:
   it's efficient but we could reduce these 2 merged into one given they are just used to compute process load, by adding `ProcessLoad plus(final ProcessLoad other)` in `ProcessLoad` class, which makes the call 1-2% faster, and arguably cleaner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "github-actions[bot] (via GitHub) via jira" <ji...@kafka.apache.org>.
github-actions[bot] commented on PR #23969:
URL: https://github.com/apache/kafka/pull/23969#issuecomment-6621505819

   A label of 'needs-attention' was automatically added to this PR in order to raise the
   attention of the committers. Once this issue has been triaged, the `triage` label
   should be removed to prevent this automation from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4605947660


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1153,11 +1118,23 @@ private static boolean isStateful(
      *        that no role it has been granted accounts for. How far behind that state is cannot be measured, because
      *        a member reports an end offset only for a task it is restoring. A process with no such task has no
      *        entry.
+     * @param processLoad
+     *        How loaded each process running at least one member is, so that the derivation can send work to the
+     *        process with the most room: the case analysis picks between equally good copies to promote, and the

Review Comment:
   agree, yea I usually let it write comment as generic as possible if I catch its defensive comments, but guess I didn't look at function signature too carefully



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4622923199


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for

Review Comment:
   it means the fields are private and only reachable through getter methods, I'll remove them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4622977656


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for
+     * one derivation, on one thread.
+     */
+    static final class TargetStandbyHolders {
+
+        private final Map<String, TasksTuple> targetAssignment;
+        private final SortedMap<String, ConfiguredSubtopology> subtopologies;
+        private Map<TaskId, Set<String>> holders;
+
+        TargetStandbyHolders(
+            final Map<String, TasksTuple> targetAssignment,
+            final SortedMap<String, ConfiguredSubtopology> subtopologies
+        ) {
+            this.targetAssignment = targetAssignment;
+            this.subtopologies = subtopologies;
+        }
+
+        /**
+         * The members the target assignment names as standby holders of the task; empty if it names none.
+         */
+        Set<String> holdersOf(final TaskId task) {
+            if (holders == null) {
+                holders = statefulStandbyHolders(targetAssignment, subtopologies);

Review Comment:
   agree



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4622935324


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target

Review Comment:
   yea I usually asked claude to avoid these for in-line comments, didn't catch the function signatures. Will clean them up



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "mjsax (via GitHub) via jira" <ji...@kafka.apache.org>.
mjsax commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4594348595


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for
+     * one derivation, on one thread.
+     */
+    static final class TargetStandbyHolders {
+
+        private final Map<String, TasksTuple> targetAssignment;
+        private final SortedMap<String, ConfiguredSubtopology> subtopologies;
+        private Map<TaskId, Set<String>> holders;
+
+        TargetStandbyHolders(
+            final Map<String, TasksTuple> targetAssignment,
+            final SortedMap<String, ConfiguredSubtopology> subtopologies
+        ) {
+            this.targetAssignment = targetAssignment;
+            this.subtopologies = subtopologies;
+        }
+
+        /**
+         * The members the target assignment names as standby holders of the task; empty if it names none.
+         */
+        Set<String> holdersOf(final TaskId task) {
+            if (holders == null) {
+                holders = statefulStandbyHolders(targetAssignment, subtopologies);

Review Comment:
   Given that we embed this one inside `TargetStandbyHolders`, it would be good to move the existing helper method into `TargetStandbyHolders` itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "mjsax (via GitHub) via jira" <ji...@kafka.apache.org>.
mjsax commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4594300295


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -117,10 +119,15 @@ static CurrentAssignmentIndex indexCurrentAssignment(
         final Map<TaskId, List<TaskCopy>> taskCopies = new HashMap<>();
         final Map<String, Set<TaskId>> reportedByProcess = new HashMap<>();
         final Map<String, Set<TaskId>> heldByProcess = new HashMap<>();
+        final Map<String, Integer> memberCounts = new HashMap<>();
+        final Map<String, Integer> statefulTaskCounts = new HashMap<>();
 
         for (final StreamsGroupMember member : members.values()) {
             final MemberTaskOffsets offsets = taskOffsets.getOrDefault(member.memberId(), MemberTaskOffsets.EMPTY);
             final Set<TaskId> heldOnProcess = heldByProcess.computeIfAbsent(member.processId(), __ -> new HashSet<>());
+            // Counted rather than read off heldOnProcess, so that the load does not depend on the invariant that
+            // a process never holds two copies of one task.
+            final int[] statefulTaskCount = new int[1];

Review Comment:
   I seems you only use a array to get some "container" that you can pass into the lambda? -- This' not really obvious -- wondering if we should add a comment, or have some small helper class
   ```
   private static final class Counter {
     private int count;
   }
   ```
   instead, which might make it more clear?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "mjsax (via GitHub) via jira" <ji...@kafka.apache.org>.
mjsax commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4594313777


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -158,6 +168,9 @@ static CurrentAssignmentIndex indexCurrentAssignment(
                 subtopologies,
                 task -> reportedByProcess.computeIfAbsent(member.processId(), __ -> new HashSet<>()).add(task)
             );
+
+            memberCounts.merge(member.processId(), 1, Integer::sum);
+            statefulTaskCounts.merge(member.processId(), statefulTaskCount[0], Integer::sum);

Review Comment:
   We have similar code through the refiner, and I am wondering if using `merge` with `Integer::sum` is really efficient code? Did not read the full PR yet, not the second PR, so maybe it's something you do later? Or did we benchmark this already and know we cannot gain anything if we would rewrite. Just curious. Ok to leave as-if on this commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4620219914


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -117,10 +119,15 @@ static CurrentAssignmentIndex indexCurrentAssignment(
         final Map<TaskId, List<TaskCopy>> taskCopies = new HashMap<>();
         final Map<String, Set<TaskId>> reportedByProcess = new HashMap<>();
         final Map<String, Set<TaskId>> heldByProcess = new HashMap<>();
+        final Map<String, Integer> memberCounts = new HashMap<>();
+        final Map<String, Integer> statefulTaskCounts = new HashMap<>();
 
         for (final StreamsGroupMember member : members.values()) {
             final MemberTaskOffsets offsets = taskOffsets.getOrDefault(member.memberId(), MemberTaskOffsets.EMPTY);
             final Set<TaskId> heldOnProcess = heldByProcess.computeIfAbsent(member.processId(), __ -> new HashSet<>());
+            // Counted rather than read off heldOnProcess, so that the load does not depend on the invariant that
+            // a process never holds two copies of one task.
+            final int[] statefulTaskCount = new int[1];

Review Comment:
   agree, I think we can just use Kafka's built in PrimitiveRef.IntRef



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4622976303


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for
+     * one derivation, on one thread.
+     */
+    static final class TargetStandbyHolders {
+
+        private final Map<String, TasksTuple> targetAssignment;
+        private final SortedMap<String, ConfiguredSubtopology> subtopologies;
+        private Map<TaskId, Set<String>> holders;
+
+        TargetStandbyHolders(
+            final Map<String, TasksTuple> targetAssignment,
+            final SortedMap<String, ConfiguredSubtopology> subtopologies
+        ) {
+            this.targetAssignment = targetAssignment;
+            this.subtopologies = subtopologies;
+        }
+
+        /**
+         * The members the target assignment names as standby holders of the task; empty if it names none.
+         */
+        Set<String> holdersOf(final TaskId task) {
+            if (holders == null) {

Review Comment:
   yes this lazy computation could save 16-41% on benchmark, I let claude ran it against 20,000 groups and 57% of them would build the map.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "mjsax (via GitHub) via jira" <ji...@kafka.apache.org>.
mjsax commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4594411135


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1153,11 +1118,23 @@ private static boolean isStateful(
      *        that no role it has been granted accounts for. How far behind that state is cannot be measured, because
      *        a member reports an end offset only for a task it is restoring. A process with no such task has no
      *        entry.
+     * @param processLoad
+     *        How loaded each process running at least one member is, so that the derivation can send work to the

Review Comment:
   > running at least one member
   
   How can a process run zero members? Of course, client side, all StreamsThreads could die, leaving the `KafkaStreams` client as "zombie". But if all members died, nobody would heartbeat and the whole process would disappear GC side, right?



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for

Review Comment:
   > The lookup is reachable only through {@link #holdersOf}, so every caller sees it built
   
   Not sure what this means, or what the point of this comment is?



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for
+     * one derivation, on one thread.
+     */
+    static final class TargetStandbyHolders {
+
+        private final Map<String, TasksTuple> targetAssignment;
+        private final SortedMap<String, ConfiguredSubtopology> subtopologies;
+        private Map<TaskId, Set<String>> holders;
+
+        TargetStandbyHolders(
+            final Map<String, TasksTuple> targetAssignment,
+            final SortedMap<String, ConfiguredSubtopology> subtopologies
+        ) {
+            this.targetAssignment = targetAssignment;
+            this.subtopologies = subtopologies;
+        }
+
+        /**
+         * The members the target assignment names as standby holders of the task; empty if it names none.
+         */
+        Set<String> holdersOf(final TaskId task) {
+            if (holders == null) {

Review Comment:
   Why don't we compute `holders` upfront in the constructor? This way, we don't need to keep `targetAssignment` and `sub-topologies`?
   
   Is this some kind of "lazy computation" optimization? If yes, would be worth a comment.



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for
+     * one derivation, on one thread.
+     */
+    static final class TargetStandbyHolders {
+
+        private final Map<String, TasksTuple> targetAssignment;
+        private final SortedMap<String, ConfiguredSubtopology> subtopologies;
+        private Map<TaskId, Set<String>> holders;
+
+        TargetStandbyHolders(
+            final Map<String, TasksTuple> targetAssignment,
+            final SortedMap<String, ConfiguredSubtopology> subtopologies
+        ) {
+            this.targetAssignment = targetAssignment;
+            this.subtopologies = subtopologies;
+        }
+
+        /**
+         * The members the target assignment names as standby holders of the task; empty if it names none.
+         */
+        Set<String> holdersOf(final TaskId task) {
+            if (holders == null) {

Review Comment:
   Curious how much this buys us? Do we have numbers in how many refiner() runs, we don't compute it with this optimization?



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find

Review Comment:
   > Only a task that diverges from its target owner reads it
   
   Why is his important? This is a helper class, and is should not depend on how it's used, and should not describe how it's used itself -- if this is important, the caller should explain it, not the callee.



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find
+     * the placement a borrowed migration holds back -- so a derivation of a group that has converged never pays for
+     * the walk. The lookup is reachable only through {@link #holdersOf}, so every caller sees it built. It lives for

Review Comment:
   > It lives for one derivation, on one thread.
   
   Similar.



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target

Review Comment:
   > built from the target assignment on the first {@link #holdersOf} call
   
   Yes, this is what the code does -- why do we have this comment? Describing what the code does, is not useful -- I can read the code :)
   
   Comments should only explain stuff that is hard to extract from reading the code, and explain the _why_, and the _what_ only in rare occasions (the code itself is the _what_, no need to repeat in plain English).



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1153,11 +1118,23 @@ private static boolean isStateful(
      *        that no role it has been granted accounts for. How far behind that state is cannot be measured, because
      *        a member reports an end offset only for a task it is restoring. A process with no such task has no
      *        entry.
+     * @param processLoad
+     *        How loaded each process running at least one member is, so that the derivation can send work to the
+     *        process with the most room: the case analysis picks between equally good copies to promote, and the

Review Comment:
   > the case analysis picks
   
   We should not describe how this is used... Only the caller know how it's used, and this `record` should be agnostic to it's usage... The description should only describe what `processLoad` represent. Nothing more.
   
   (It's a weird Claude trait to add comments like this -- I added a rule in my CLAUDE.md to it stops doing this...)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4622939432


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1411,6 +1388,40 @@ private record StandbyConflicts(
     ) {
     }
 
+    /**
+     * The members the target assignment names as standby holders of each stateful task.
+     *
+     * <p>The lookup is built from the target assignment on the first {@link #holdersOf} call, which walks every target
+     * standby. Only a task that diverges from its target owner reads it -- to rank the copies to promote, and to find

Review Comment:
   agree



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-21273: Make each streams assignment refiner call cheaper [kafka]

Posted by "suzhiking (via GitHub) via jira" <ji...@kafka.apache.org>.
suzhiking commented on code in PR #23969:
URL: https://github.com/apache/kafka/pull/23969#discussion_r4623079781


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java:
##########
@@ -1153,11 +1118,23 @@ private static boolean isStateful(
      *        that no role it has been granted accounts for. How far behind that state is cannot be measured, because
      *        a member reports an end offset only for a task it is restoring. A process with no such task has no
      *        entry.
+     * @param processLoad
+     *        How loaded each process running at least one member is, so that the derivation can send work to the

Review Comment:
   uh I did not know this, I think it's actually from your own commit https://github.com/apache/kafka/blob/30a33d5a8bc8c8d6d41516b9e7113dc822b867be/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/AssignmentRefinerImpl.java#L185-L208. I'll trim them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org