[jira] [Created] (KAFKA-12675) Improve sticky general assignor scalability and performance

["Luke Chen (Jira)" <ji...@apache.org>]

Luke Chen created KAFKA-12675:
---------------------------------

             Summary: Improve sticky general assignor scalability and performance
                 Key: KAFKA-12675
                 URL: https://issues.apache.org/jira/browse/KAFKA-12675
             Project: Kafka
          Issue Type: Improvement
            Reporter: Luke Chen
            Assignee: Luke Chen


Currently, we have "general assignor" for non-equal subscription case and "constrained assignor" for all equal subscription case. There's a performance test for constrained assignor with:

topicCount = {color:#0000ff}500{color};
partitionCount = {color:#0000ff}2000{color}; 
consumerCount = {color:#0000ff}2000{color};

in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million partitions and we can complete the assignment within 2 second in my machine.

However, if we let 1 of the consumer subscribe to only 1 topic, it'll use "general assignor", and the result with the same setting as above is: *OutOfMemory,* 
Even we down the count to:

topicCount = {color:#0000ff}50{color};
partitionCount = 1{color:#0000ff}000{color}; 
consumerCount = 1{color:#0000ff}000{color};

We still got *OutOfMemory*.

With this setting:

topicCount = {color:#0000ff}50{color};
partitionCount = 8{color:#0000ff}00{color}; 
consumerCount = 8{color:#0000ff}00{color};

We can complete with 10 seconds in my machine, which is still slow.

 

Since we are going to set default assignment strategy to "CooperativeStickyAssignor" soon,  we should improve the scalability and performance for sticky general assignor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)