You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Sophie Blee-Goldman (Jira)" <ji...@apache.org> on 2020/05/14 00:38:00 UTC

[jira] [Updated] (KAFKA-9987) Improve sticky partition assignor algorithm

     [ https://issues.apache.org/jira/browse/KAFKA-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sophie Blee-Goldman updated KAFKA-9987:
---------------------------------------
    Summary: Improve sticky partition assignor algorithm  (was: Add new cooperative assignor optimized for constant-subscription group)

> Improve sticky partition assignor algorithm
> -------------------------------------------
>
>                 Key: KAFKA-9987
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9987
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients
>            Reporter: Sophie Blee-Goldman
>            Assignee: Sophie Blee-Goldman
>            Priority: Major
>
> In KIP-429 we added the new CooperativeStickyAssignor which leverages on the underlying sticky assignment algorithm of the existing StickyAssignor (moved to AbstractStickyAssignor). 
>  
> The algorithm is fairly complex as it tries to optimize stickiness while satisfying perfect balance _in the case individual consumers may be subscribed to a random subset of the topics._ While it does a pretty good job at what it promises to do, it doesn't scale well with large numbers of consumers and partitions. 
>  
> If we can make the assumption that all consumers are subscribed to the same set of topics, we can simplify the algorithm greatly and do a sticky-but-balanced assignment in a single pass. It would be nice to have an additional cooperative assignor OOTB that performs efficiently for users who know their group will satisfy this constraint.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)