You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Sophie Blee-Goldman (Jira)" <ji...@apache.org> on 2020/05/13 16:52:00 UTC
[jira] [Updated] (KAFKA-9987) Add new cooperative assignor
optimized for constant-subscription group
[ https://issues.apache.org/jira/browse/KAFKA-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sophie Blee-Goldman updated KAFKA-9987:
---------------------------------------
Summary: Add new cooperative assignor optimized for constant-subscription group (was: Add new cooperative assignor optimized for constant-subscription. group)
> Add new cooperative assignor optimized for constant-subscription group
> ----------------------------------------------------------------------
>
> Key: KAFKA-9987
> URL: https://issues.apache.org/jira/browse/KAFKA-9987
> Project: Kafka
> Issue Type: Improvement
> Components: clients
> Reporter: Sophie Blee-Goldman
> Assignee: Sophie Blee-Goldman
> Priority: Major
>
> In KIP-429 we added the new CooperativeStickyAssignor which leverages on the underlying sticky assignment algorithm of the existing StickyAssignor (moved to AbstractStickyAssignor).
>
> The algorithm is fairly complex as it tries to optimize stickiness while satisfying perfect balance _in the case individual consumers may be subscribed to a random subset of the topics._ While it does a pretty good job at what it promises to do, it doesn't scale well with large numbers of consumers and partitions.
>
> If we can make the assumption that all consumers are subscribed to the same set of topics, we can simplify the algorithm greatly and do a sticky-but-balanced assignment in a single pass. It would be nice to have an additional cooperative assignor OOTB that performs efficiently for users who know their group will satisfy this constraint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)