[jira] [Commented] (KAFKA-7690) Change disk allocation policy for multiple partitions on a broker when topic is created

["haiyangyu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704378#comment-16704378 ] 

haiyangyu commented on KAFKA-7690:
----------------------------------

[~omkreddy]

[~ijuma]

[~guozhang]

[~hachikuji]

[~cmccabe]

[~junrao]

> Change disk allocation policy for multiple partitions on a broker when topic is created
> ---------------------------------------------------------------------------------------
>
>                 Key: KAFKA-7690
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7690
>             Project: Kafka
>          Issue Type: Improvement
>    Affects Versions: 0.10.2.0, 1.0.0, 2.0.0
>            Reporter: haiyangyu
>            Priority: Major
>
> h3. *Background*
> if target topic partitions lager than broker size when create a topic or add partition, one broker will be assigned more than one partition. if current all disk is not balance, such as one disk has one partition and the other one has four partitions due to topic delete or others, the mutil partitions will be all allocated in the first disk, and if the target topic has a large flow, it is easily to fill up the disk io.
> h3. *Improvement strategy*
> when mutil ** partition is going to be allocated on a broker, the strategy is as follow:
> 1、calculate the target topic partition count and total partition count on each disk.
> topic count
> 2、sorted by the target topic partition count wich ascending order, if the target topic partition count is equal, sorted by the total partitions on each disk.
> h3. *Example*
>  
> ||disk||target topic partition count||total partition count||
> |disk1|0|11|
> |disk2|0|9|
> when tow partitions are assigned on this broker, if use origin strategy, the result is as follows:
> ||disk||target topic partition count||total partition count||
> |disk1|0|11|
> |disk2|2|11|
> use new strategy, the result is as follows:
> ||disk||target topic partition count||total partition count||
> |disk1|1|12|
> |disk2|1|10|
> if the topic has a huge flow such as 50MB/s per partition, it is easily to fill up disk2 io if use origin strategy. However if use new strategy, it's well in disk io rebalance.
> h3. *Summary*
> This strategy is good to build a big cluster and create a topic which has huge amount partition, the disk io will be more balanced.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)