You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Wang Xixu (Code Review)" <ge...@cloudera.org> on 2022/09/07 11:58:50 UTC
[kudu-CR] [Tools] Relance data in different disks according to container number
Wang Xixu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18946
Change subject: [Tools] Relance data in different disks according to container number
......................................................................
[Tools] Relance data in different disks according to container number
There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more some new disks, Kudu can not
rebalance the data in disks immediately.
Therefore, this patch rebalances the data in disks offline. It will statistics
the number of containers of diffrent disks. Rebalance the data according to the
number of containers. Only log-block type is supported.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/block_manager.h
M src/kudu/fs/file_block_manager.cc
M src/kudu/fs/fs-test-util.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
8 files changed, 254 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/1
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 1
Gerrit-Owner: Wang Xixu <14...@qq.com>
[kudu-CR] [Tools] Relance data in different disks according to container number
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#3).
Change subject: [Tools] Relance data in different disks according to container number
......................................................................
[Tools] Relance data in different disks according to container number
There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more some new disks, Kudu can not
rebalance the data in disks immediately.
Therefore, this patch rebalances the data in disks offline. It will statistics
the number of containers of diffrent disks. Rebalance the data according to the
number of containers. Only log-block type is supported.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
4 files changed, 229 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/3
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 3
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [Tools] Rebalance data among different disks
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/18946 )
Change subject: [Tools] Rebalance data among different disks
......................................................................
Patch Set 6:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/fs/file_block_manager.cc
File src/kudu/fs/file_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/fs/file_block_manager.cc@696
PS6, Line 696: Status FileBlockManager::MigrateBlock(CreateBlockOptions opts,
> warning: parameter 'opts' is unused [misc-unused-parameters]
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/fs/file_block_manager.cc@697
PS6, Line 697: const BlockId& block_id) {
> warning: parameter 'block_id' is unused [misc-unused-parameters]
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/fs/file_block_manager.cc@701
PS6, Line 701: Status FileBlockManager::GetBlockDirIdx(const BlockId& block_id, int* dir) {
> warning: parameter 'block_id' is unused [misc-unused-parameters]
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/fs/log_block_manager.cc@2723
PS6, Line 2723: LogBlockRefPtr LogBlockManager::CreateLogBlock(
> warning: method 'CreateLogBlock' can be made static [readability-convert-me
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_server-test.cc
File src/kudu/tserver/tablet_server-test.cc:
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_server-test.cc@3858
PS6, Line 3858: virtual void SetUp() OVERRIDE {
> warning: 'virtual' is redundant since the function is already declared 'ove
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_service.h
File src/kudu/tserver/tablet_service.h:
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_service.h@97
PS6, Line 97: class DiskRebalanceResponsePB
> error: expected ';' after class [clang-diagnostic-error]
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_service.h@243
PS6, Line 243: void DiskRebalance(const DiskRebalanceRequestPB* req,
> warning: annotate this function with 'override' or (rarely) 'final' [modern
Done
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_service.cc
File src/kudu/tserver/tablet_service.cc:
http://gerrit.cloudera.org:8080/#/c/18946/6/src/kudu/tserver/tablet_service.cc@1455
PS6, Line 1455: void TabletServiceAdminImpl::DiskRebalance(const DiskRebalanceRequestPB* req,
> warning: parameter 'req' is unused [misc-unused-parameters]
Done
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 6
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Comment-Date: Tue, 20 Dec 2022 06:18:28 +0000
Gerrit-HasComments: Yes
[kudu-CR] [Tools] Relance data in different disks according to container number
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#2).
Change subject: [Tools] Relance data in different disks according to container number
......................................................................
[Tools] Relance data in different disks according to container number
There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more some new disks, Kudu can not
rebalance the data in disks immediately.
Therefore, this patch rebalances the data in disks offline. It will statistics
the number of containers of diffrent disks. Rebalance the data according to the
number of containers. Only log-block type is supported.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
4 files changed, 213 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/2
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [Tools] Rebalance data among different disks
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#6).
Change subject: [Tools] Rebalance data among different disks
......................................................................
[Tools] Rebalance data among different disks
There already existed a good algorithm to make data distribute
in different disks balancely. But when a host adding more new
disks, Kudu can not rebalance the data among disks immediately.
This patch offer a tool to rebalance data among different disks.
It firstly statistics of the number of tablets in disks. Then
move the tablets from the disk owning most tablets to the less
one. In reality, it migrates blocks from one disk to the other.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/block_manager.h
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/data_dirs.h
M src/kudu/fs/file_block_manager.cc
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tablet_service.h
M src/kudu/tserver/tserver_admin.proto
12 files changed, 509 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/6
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 6
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
[kudu-CR] [Tools] Relance data in different disks according to container size
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/18946 )
Change subject: [Tools] Relance data in different disks according to container size
......................................................................
Patch Set 5:
> Patch Set 4:
>
> Thanks for the patch. If the containers are not of the same size (which is typically the case in real clusters) this approach might not be the ideal way to balance the data. This approach approach could make the consumption more imbalanced than already if the skewness in the size of the containers is high.
> In addition to the container count, can we get the size of the containers and balance so that the disk consumption is taken into account.
Thanks. The factor of disk consumption of container is more available than container number.
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Comment-Date: Fri, 16 Sep 2022 03:17:30 +0000
Gerrit-HasComments: No
[kudu-CR] [Tools] Rebalance data among different disks
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#7).
Change subject: [Tools] Rebalance data among different disks
......................................................................
[Tools] Rebalance data among different disks
There already existed a good algorithm to make data distribute
in different disks balancely. But when a host adding more new
disks, Kudu can not rebalance the data among disks immediately.
This patch offer a tool to rebalance data among different disks.
It firstly statistics of the number of tablets in disks. Then
move the tablets from the disk owning most tablets to the less
one. In reality, it migrates blocks from one disk to the other.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/block_manager.h
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/data_dirs.h
M src/kudu/fs/file_block_manager.cc
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tablet_service.h
M src/kudu/tserver/tserver_admin.proto
12 files changed, 509 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/7
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 7
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
[kudu-CR] [Tools] Relance data in different disks according to container number
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#4).
Change subject: [Tools] Relance data in different disks according to container number
......................................................................
[Tools] Relance data in different disks according to container number
There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more some new disks, Kudu can not
rebalance the data in disks immediately.
Therefore, this patch rebalances the data in disks offline. It will statistics
the number of containers of diffrent disks. Rebalance the data according to the
number of containers. Only log-block type is supported.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
4 files changed, 221 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/4
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 4
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [Tools] Relance data in different disks according to container size
Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#5).
Change subject: [Tools] Relance data in different disks according to container size
......................................................................
[Tools] Relance data in different disks according to container size
There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more new disks, Kudu can not
rebalance the data in disks immediately.
Therefore, this patch rebalances the data in disks offline. It will statistics
the disk consumption of containers of different disks. Rebalance the data according
to the file size of containers. Only log-block type is supported.
The algorithm is simple. Sort disks accroding to the size of all log block containers.
Travel the sorted disk, try to move a log block container from highest consumption disk
to lowest one, util the highest is less or equal than the average size, or the least is
more or equal than the average size, or can not find a suitable log block container
to be moved. This algorithm can not achieve totally balance once.
Why not take tablet as the minimum uint of moving data? A tablet may be distributed in
different disks controled by parameter FLAGS_fs_target_data_dirs_per_tablet. It needs to
remap all tablet in different disks. Log block is also unsuitable to taken as the minimun
unit. Log block is not existed as a unique file. Moving a log block is difficult. It needs
to update all metadata in superblock. Log block container has no status and it only contains
two file: .data file and .metadata file. It is easy to achieve file moving.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
4 files changed, 263 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/5
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [Tools] Relance data in different disks according to container number
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/18946 )
Change subject: [Tools] Relance data in different disks according to container number
......................................................................
Patch Set 4:
Thanks for the patch. If the containers are not of the same size (which is typically the case in real clusters) this approach might not be the ideal way to balance the data. This approach approach could make the consumption more imbalanced than already if the skewness in the size of the containers is high.
In addition to the container count, can we get the size of the containers and balance so that the disk consumption is taken into account.
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 4
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 14 Sep 2022 02:53:31 +0000
Gerrit-HasComments: No