[jira] [Commented] (KAFKA-10888) Sticky partition leads to uneven product msg, resulting in abnormal delays in some partations

["Jason Gustafson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285383#comment-17285383 ] 

Jason Gustafson commented on KAFKA-10888:
-----------------------------------------

We have found one cause of imbalance when the sticky partitioner is used. Basically the intuition behind the sticky partitioner breaks down a little bit when a small `linger.ms` is in use (sadly this is the default). The user is sort of opting out of batching with this setting which means there is little opportunity to fill batches before they get drained and sent. That leaves the door open for a kind of imbalanced write problem.

To see why, suppose that we have a producer writing to 3 partitions with linger.ms=0 and one partition slows down a little bit for some reason. It could be a leader change or some transient network issue. The producer will have to hold onto the batches for that partition until it becomes available. While it is holding onto those batches, additional batches will begin piling up. Each of these batches is likely to get filled because the producer is not ready to send to this partition yet.

Consider this from the perspective of the sticky partitioner. Every time the slow partition gets selected, the producer will fill the batches completely. On the other hand, the remaining "fast" partitions will likely not get their batches filled because of the `linger.ms=0` setting. As soon as a single record is available, it might get sent. So more data ends up getting written to the partition that has already started to build a backlog. And even after the cause of the original slowness (e.g. leader change) gets resolved, it might take some time for this imbalance to recover. We believe this can even create a runaway effect if the partition cannot catch up with the handicap of the additional load.

We analyzed one case where we thought this might be going on. Below I've summarized the writes over a period of one hour to 3 partitions. Partition 0 here is the "slow" partition. All partitions get roughly the same number of batches, but the slow partition has much bigger batch sizes.

{code}
Partition TotalBatches TotalBytes TotalRecords BytesPerBatch RecordsPerBatch
0         1683         25953200   25228        15420.80      14.99        
1         1713         7836878    4622         4574.94       2.70
2         1711         7546212    4381         4410.41       2.56
{code}

After restarting the application, the producer was healthy again. It just was not able to recover with the imbalanced workload.

>  Sticky partition leads to uneven product msg, resulting in abnormal delays in some partations
> ----------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-10888
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10888
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients, producer 
>    Affects Versions: 2.4.1
>            Reporter: jr
>            Priority: Major
>         Attachments: image-2020-12-24-21-05-02-800.png, image-2020-12-24-21-09-47-692.png, image-2020-12-24-21-10-24-407.png
>
>
>   110 producers ,550 partitions ,550 consumers , 5 nodes Kafka cluster
>   The producer uses the nullkey+stick partitioner, the total production rate is about 100w tps
> Observed partition delay is abnormal and message distribution is uneven, which leads to the maximum production and consumption delay of the partition with more messages 
> abnormal.
>   I cannot find reason that stick will make the message distribution uneven at this production rate.
>   I can't switch to the round-robin partitioner, which will increase the delay and cpu cost. Is thathe stick partationer design cause uneven message distribution, or this is abnormal. How to solve it?
>   !image-2020-12-24-21-09-47-692.png!
> As shown in the picture, the uneven distribution is concentrated on some partitions and some brokers, there seems to be some rules.
> This problem does not only occur in one cluster, but in many high tps clusters,
> The problem is more obvious on the test cluster we built.
> !image-2020-12-24-21-10-24-407.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)