[jira] [Commented] (KAFKA-10888) Sticky partition leads to uneven product msg, resulting in abnormal delays in some partitions

["nk2242696 (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17458258#comment-17458258 ] 

nk2242696 commented on KAFKA-10888:
-----------------------------------

[~showuon] [~hachikuji] 

I propose a solution based on keeping the  track of number of offsets/messages written to each partition. 
 # At partitioner level, Store the number of offsets/messages written to each partition. (sizePerPartitionMap) and total(total offsets for a topic)
 # Before choosing next batch to write(onNextBatch()) . Use (sizePerPartitionMap) to blacklisting the available partitions which causes skewness(USING a configurable THRESHOLD %) . Choose next partition from list of available whitelisted partitions. 
 # To configure sizePerPartitionMap. Use the callback method of producer.send() to update the sizePerPartitionMap and total .

This way, we can skip slower partitions(blacklisted) for few rounds and ensure all partitions are roughly of equal size. 

>  Sticky partition leads to uneven product msg, resulting in abnormal delays in some partitions
> ----------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-10888
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10888
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients, producer 
>    Affects Versions: 2.4.1
>            Reporter: jr
>            Assignee: Luke Chen
>            Priority: Major
>         Attachments: image-2020-12-24-21-05-02-800.png, image-2020-12-24-21-09-47-692.png, image-2020-12-24-21-10-24-407.png
>
>
>   110 producers ,550 partitions ,550 consumers , 5 nodes Kafka cluster
>   The producer uses the nullkey+stick partitioner, the total production rate is about 100w tps
> Observed partition delay is abnormal and message distribution is uneven, which leads to the maximum production and consumption delay of the partition with more messages 
> abnormal.
>   I cannot find reason that stick will make the message distribution uneven at this production rate.
>   I can't switch to the round-robin partitioner, which will increase the delay and cpu cost. Is thathe stick partationer design cause uneven message distribution, or this is abnormal. How to solve it?
>   !image-2020-12-24-21-09-47-692.png!
> As shown in the picture, the uneven distribution is concentrated on some partitions and some brokers, there seems to be some rules.
> This problem does not only occur in one cluster, but in many high tps clusters,
> The problem is more obvious on the test cluster we built.
> !image-2020-12-24-21-10-24-407.png!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)