You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Wang Xixu (Code Review)" <ge...@cloudera.org> on 2022/09/16 03:16:02 UTC

[kudu-CR] [Tools] Relance data in different disks according to container size

Hello Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18946

to look at the new patch set (#5).

Change subject: [Tools] Relance data in different disks according to container size
......................................................................

[Tools] Relance data in different disks according to container size

There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more new disks, Kudu can not
rebalance the data in disks immediately.

Therefore, this patch rebalances the data in disks offline. It will statistics
the disk consumption of containers of different disks. Rebalance the data according
to the file size of containers. Only log-block type is supported.

The algorithm is simple. Sort disks accroding to the size of all log block containers.
Travel the sorted disk, try to move a log block container from highest consumption disk
to lowest one, util the highest is less or equal than the average size, or the least is
more or equal than the average size, or can not find a suitable log block container
to be moved. This algorithm can not achieve totally balance once.

Why not take tablet as the minimum uint of moving data? A tablet may be distributed in
different disks controled by parameter FLAGS_fs_target_data_dirs_per_tablet. It needs to
remap all tablet in different disks. Log block is also unsuitable to taken as the minimun
unit. Log block is not existed as a unique file. Moving a log block is difficult. It needs
to update all metadata in superblock. Log block container has no status and it only contains
two file: .data file and .metadata file. It is easy to achieve file moving.

Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
4 files changed, 263 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)