You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/08/11 13:39:49 UTC
[GitHub] [incubator-doris] weizuo93 opened a new issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
weizuo93 opened a new issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329
When creating a tablet, it is necessary to select a disk from all disks that meet the requirements on the BE node to store the tablet. In doris, the current disk selection strategy is to randomly select a disk from all disks that meet the requirements for tablet creation. After the cluster has been running for a long time, we found that the distribution of the number of tablets on different disks in a BE node is severely unbalanced.
After analysis, we think the main reason for this problem is that the random disk selection strategy used in doris does not consider the problem of load balance between different disks on a BE node. A good disk selection strategy should take into account both the randomness of selection and disk load. In order to solve this problem, we introduced the the algorithm of "two random choices" for disk selection when creating the tablet:
(1) Select two disks from all disks that meet the requirements on the BE node randomly；
(2) Choose the disk with a smaller number of tablet from the two disks selected in (1) for tablet creation.
In the initial state, the distribution of tablets among different disks is balanced. The simulation experiments show that the distribution of tablets on different disks in the BE node always remains relatively balanced with the creation of tablets after used the "two random choices" disk selection algorithm. (The top figure shows the range(极差) trend of tablet distribution on different disks in a BE node with the creation of tablets. The bottom figure shows the standard deviation(标准差) trend of tablet distribution on different disks in a BE node with the creation of tablets. In these two figures, the red line indicates the random disk selection strategy used in doris for tablet creation and the blue line indicates "two random choices" disk selection strategy we introduced for tablet creation )
In the initial state, the distribution of tablets among different disks is unbalanced. The simulation experiments show that the "two random choice" disk selection algorithm can also gradually balance the load of different disks with the creation of tablets. (The top figure shows the range(极差) trend of tablet distribution on different disks in a BE node with the creation of tablets. The bottom figure shows the standard deviation(标准差) trend of tablet distribution on different disks in a BE node with the creation of tablets. In these two figures, the red line indicates the random disk selection strategy used in doris for tablet creation and the blue line indicates "two random choices" disk selection strategy we introduced for tablet creation )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329#issuecomment-674686945
> I think this is a good way to improve.
> In fact, there are currently two problems with disk balancing.
> One, as you described, needs to consider the current disk load when creating a tablet.
> The other is how to support tablet migration between disks on the same machine.
>
> These are two independent problems. We can solve the first problem first.
@morningman
Thanks very much for your reply. We are optimizing the disk selection strategy for tablet creation on BE nodes recently (PR #4373 ) . As you mentioned, there is another problem that tablet migration between disks on the same machine is neccessary, and we will start related work next.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329#issuecomment-674342416
I think this is a good way to improve.
In fact, there are currently two problems with disk balancing.
One, as you described, needs to consider the current disk load when creating a tablet.
The other is how to support tablet migration between disks on the same machine.
These are two independent problems. We can solve the first problem first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329#issuecomment-673859068
Doris BE节点中创建tablet时磁盘选择的策略是随机选择一个磁盘，具体实现方法是：首先，将所有满足要求(storage medium type)的磁盘存放在一个vector中；然后，使用random_shuffle()将vector中磁盘顺序打乱；最后，创建tablet时，从vector中的第一磁盘开始尝试，如果创建失败，换下一个磁盘，直到在某一个磁盘上成功创建tablet。这种随机的磁盘选择策略并没有考虑不同磁盘上tablet的分布问题，导致集群运行一段时间后，BE节点上不同磁盘上的tablet数量很可能分布不均衡。
如果采用“two random choices”策略进行tablet创建时的磁盘选择，则会有更好的效果。“two random choices”策略的核心思想是：首先，从所有满足要求(storage medium type)的磁盘中随机选择两个磁盘；然后，从随机选出的两个磁盘中选择tablet数量较少的磁盘用于tablet的创建。通过实验发现，“two random choices”策略能够更好地保证集群运行过程中，BE节点上不同磁盘上的tablet更均衡地分布；另外，当初始状态BE节点上不同磁盘的tablet分布不均衡时，two random choices”策略也能在后续随着tablet的创建而使得不同磁盘上的tablet分布逐渐趋于均衡。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 edited a comment on issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
weizuo93 edited a comment on issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329#issuecomment-673859068
Doris BE节点中创建tablet时磁盘选择的策略是随机选择一个磁盘，具体实现方法是：首先，将所有满足要求(storage medium type)的磁盘存放在一个vector中；然后，使用random_shuffle()将vector中磁盘顺序打乱；最后，创建tablet时，从vector中的第一磁盘开始尝试，如果创建失败，换下一个磁盘，直到在某一个磁盘上成功创建tablet。这种随机的磁盘选择策略并没有考虑不同磁盘上tablet的分布问题，导致集群运行一段时间后，BE节点上不同磁盘上的tablet数量很可能分布不均衡。
如果采用“two random choices”策略进行tablet创建时的磁盘选择，则会有更好的效果。“two random choices”策略的核心思想是：首先，从所有满足要求(storage medium type)的磁盘中随机选择两个磁盘；然后，从随机选出的两个磁盘中选择tablet数量较少的磁盘用于tablet的创建。通过实验发现，“two random choices”策略能够更好地保证集群运行过程中，BE节点上不同磁盘上的tablet更均衡地分布；另外，当初始状态BE节点上不同磁盘的tablet分布不均衡时，two random choices”策略也能在后续随着tablet的创建而使得不同磁盘上的tablet分布逐渐趋于均衡。
@morningman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329#issuecomment-671969261
In the simulation experiment, the number of disks on the BE node is set to 10, and the total number of tablets created after the initial state is 30000.
In the initial state, the distribution of tablets among different disks is balanced. When using different disk selection strategies, the number of tablets on different disks is distributed as shown in the figure below after all tablets are created.
In the initial state, the distribution of tablets among different disks is unbalanced. When using different disk selection strategies, the number of tablets on different disks is distributed as shown in the figure below after all tablets are created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
morningman closed issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 removed a comment on issue #4329: [Proposal] Optimize the disk selection strategy for tablet creation
Posted by GitBox <gi...@apache.org>.
weizuo93 removed a comment on issue #4329:
URL: https://github.com/apache/incubator-doris/issues/4329#issuecomment-671969261
In the simulation experiment, the number of disks on the BE node is set to 10, and the total number of tablets created after the initial state is 30000.
In the initial state, the distribution of tablets among different disks is balanced. When using different disk selection strategies, the number of tablets on different disks is distributed as shown in the figure below after all tablets are created.
In the initial state, the distribution of tablets among different disks is unbalanced. When using different disk selection strategies, the number of tablets on different disks is distributed as shown in the figure below after all tablets are created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org