[jira] [Created] (KAFKA-21308) StickyTaskAssignor: a full previous member hides other previous members that have room

["Ziyun Fu (Jira)" <ji...@apache.org>]

Ziyun Fu created KAFKA-21308:
--------------------------------

             Summary:  StickyTaskAssignor: a full previous member hides other previous members that have room 
                 Key: KAFKA-21308
                 URL: https://issues.apache.org/jira/browse/KAFKA-21308
             Project: Kafka
          Issue Type: Improvement
            Reporter: Ziyun Fu


{{StickyTaskAssignor#findPrevMemberWithLeastLoad}} picks one previous member by load, and the caller checks that member's quota only afterwards. If the picked member is at its quota, the lookup returns no member, even when another previous member of the task is below its quota. The task then goes to the least-loaded process, which may not have the task's state.

The candidate is replaced only when another member is strictly lower on both process load and member load, so a full member stays picked easily.

*Example:* task {{t}} has previous standby members {{a1}} (process A) and {{b1}} (process B), with {{a1}} first in the list.
 * {{a1}} is at the per-member quota. A's other member {{a2}} has no tasks, so A's process load is lower than B's.
 * {{b1}} is below the quota, but B's process load is not lower than A's, so {{a1}} stays picked.
 * The quota check on {{a1}} fails, and the standby of {{t}} goes to the least-loaded process (for example {{{}a2{}}}, which has no state), not to {{{}b1{}}}.

*Affected call sites:*
 * Active task assignment from previous standby members ({{{}hasUnfulfilledActiveTaskQuota{}}} checked after the lookup)
 * Standby task assignment from previous standby members ({{{}hasUnfulfilledTaskQuota{}}} checked after the lookup)
 * The rack-aware standby steps of KAFKA-20999, which use the same lookup

*Proposed fix:* check the quota inside {{{}findPrevMemberWithLeastLoad{}}}, skipping members that are at their quota, so the least-loaded previous member with room is picked. This changes some assignments, since more tasks stay on members that have their state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)