[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance

["Luke Chen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322599#comment-17322599 ] 

Luke Chen edited comment on KAFKA-12675 at 4/16/21, 3:59 AM:
-------------------------------------------------------------

[~twmb], thanks for suggestion. But in this ticket, I'd like to focus on improving the scalability and performance first. I agree the whole algorithm can be improved, but that needs to go through KIP and have more discussion to be able to go on. Thanks anyway, I'll think about it. :)


was (Author: showuon):
[~twmb], thanks for suggestion. But in this ticket, I'd like to improve the scalability and performance first. I agree the whole algorithm can be improved, too, but that needs to go through KIP and have more discussion to be able to go on. Thanks anyway, I'll think about it. :)

> Improve sticky general assignor scalability and performance
> -----------------------------------------------------------
>
>                 Key: KAFKA-12675
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12675
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Luke Chen
>            Assignee: Luke Chen
>            Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and "constrained assignor" for all equal subscription case. There's a performance test for constrained assignor with:
> topicCount = {color:#0000ff}500{color};
>  partitionCount = {color:#0000ff}2000{color}; 
>  consumerCount = {color:#0000ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use "general assignor", and the result with the same setting as above is: *OutOfMemory,* 
>  Even we down the count to:
> topicCount = {color:#0000ff}50{color};
>  partitionCount = 1{color:#0000ff}000{color}; 
>  consumerCount = 1{color:#0000ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#0000ff}50{color};
>  partitionCount = 8{color:#0000ff}00{color}; 
>  consumerCount = 8{color:#0000ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>  
> Since we are going to set default assignment strategy to "CooperativeStickyAssignor" soon,  we should improve the scalability and performance for sticky general assignor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)