[jira] [Commented] (KAFKA-12675) Improve sticky general assignor scalability and performance

["Travis Bischel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322593#comment-17322593 ] 

Travis Bischel commented on KAFKA-12675:
----------------------------------------

An option to evaluate is the algorithm I devised in my franz-go client, which translates the balancing into a graph and uses A* search to perform an exact balance much more efficiently. I noticed that the existing Java algorithm is heuristic based, and I have a few tests in my repo showing edge cases that the existing heuristic algorithm cannot really handle.

The algorithm is here: https://github.com/twmb/franz-go/blob/master/pkg/kgo/internal/sticky/graph.go
with the option to switch into that algorithm in the sticky.go file.

> Improve sticky general assignor scalability and performance
> -----------------------------------------------------------
>
>                 Key: KAFKA-12675
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12675
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Luke Chen
>            Assignee: Luke Chen
>            Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and "constrained assignor" for all equal subscription case. There's a performance test for constrained assignor with:
> topicCount = {color:#0000ff}500{color};
>  partitionCount = {color:#0000ff}2000{color}; 
>  consumerCount = {color:#0000ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use "general assignor", and the result with the same setting as above is: *OutOfMemory,* 
>  Even we down the count to:
> topicCount = {color:#0000ff}50{color};
>  partitionCount = 1{color:#0000ff}000{color}; 
>  consumerCount = 1{color:#0000ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#0000ff}50{color};
>  partitionCount = 8{color:#0000ff}00{color}; 
>  consumerCount = 8{color:#0000ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>  
> Since we are going to set default assignment strategy to "CooperativeStickyAssignor" soon,  we should improve the scalability and performance for sticky general assignor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)