You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Gabriella Lotz (Code Review)" <ge...@cloudera.org> on 2026/10/02 11:41:25 UTC
[kudu-CR] [auto rebalancer] enforce per-server move cap
Gabriella Lotz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/25005
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
[auto_rebalancer] enforce per-server move cap
The auto-rebalancer never enforced its per-server move cap. It budgeted
a whole batch at max_moves_per_server * num_tservers and checked
moves_per_tserver_, which is empty at selection time, so one batch could
pile many moves onto a single server (the most loaded one, which the cap
exists to protect).
Track per-server move counts within the batch and skip a move once
either endpoint reaches the cap.
Add MaxMovesPerServerEnforcedWithinOneBatch, which is red before this
change: a 6-move batch puts 3 moves on one server with the cap at 1.
Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
---
M src/kudu/master/auto_rebalancer-test.cc
M src/kudu/master/auto_rebalancer.cc
2 files changed, 110 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/25005/1
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 5:
> Patch Set 4: Code-Review+2
>
> LGTM.
>
> Please make sure the pre-commit test failures are unrelated to this changelist.
I noticed AddTserver in auto_leader_rebalancer-test.cc failed on the debug and release builds. It turned out the test is too slow now that the per-server cap is enforced, so I raised auto_rebalancing_max_moves_per_server to 5 in the test fixture. Without that change AddTserver failed locally, and with it, it passed 20 out of 20 repeated runs in a debug build.
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 14:08:49 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Alexey Serbin, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25005
to look at the new patch set (#2).
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
[auto_rebalancer] enforce per-server move cap
The auto-rebalancer never enforced its per-server move cap. It budgeted
a whole batch at max_moves_per_server * num_tservers and checked
moves_per_tserver_, which is empty at selection time, so one batch could
pile many moves onto a single server (the most loaded one, which the cap
exists to protect).
Track per-server move counts within the batch and skip a move once
either endpoint reaches the cap.
Add MaxMovesPerServerEnforcedWithinOneBatch, which is red before this
change: a 6-move batch puts 3 moves on one server with the cap at 1.
Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
---
M src/kudu/master/auto_rebalancer-test.cc
M src/kudu/master/auto_rebalancer.cc
2 files changed, 114 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/25005/2
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3076/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 02 Oct 2026 12:24:12 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3091/
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 14:06:28 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 4:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3133/
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Oct 2026 12:00:38 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/25005/1/src/kudu/master/auto_rebalancer.cc
File src/kudu/master/auto_rebalancer.cc:
http://gerrit.cloudera.org:8080/#/c/25005/1/src/kudu/master/auto_rebalancer.cc@488
PS1, Line 488: RETURN_NOT_OK(rebalancer_.FilterCrossLocationTabletCandidates(
> For a move that SelectReplicaToMove turns into a pure drop (replication_fac
It is intended as conservative behavior. I have added a comment explaining that.
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 14:06:30 +0000
Gerrit-HasComments: Yes
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 5:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3142/
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 14:08:00 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/25005/1/src/kudu/master/auto_rebalancer.cc
File src/kudu/master/auto_rebalancer.cc:
http://gerrit.cloudera.org:8080/#/c/25005/1/src/kudu/master/auto_rebalancer.cc@488
PS1, Line 488: RETURN_NOT_OK(SelectReplicaToMove(move, extra_info_by_tablet_id,
For a move that SelectReplicaToMove turns into a pure drop (replication_factor < num_voters -> ts_uuid_to cleared), the destination move.to is checked against the cap here but is never incremented below. So a drop can be skipped because move.to looks "full," even though executing a drop adds nothing to move.to. The check (move.to) and the increment (selected.ts_uuid_to, conditionally) are asymmetric for that path.
Question: is gating drops on move.to intended as conservative behavior, or should the cap only consider the source for drop moves? It's safe either way (worst case a drop is deferred to the next round), so this is just asking you to confirm intent - a one-line comment would settle it.
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 15:42:07 +0000
Gerrit-HasComments: Yes
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Alexey Serbin, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25005
to look at the new patch set (#5).
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
[auto_rebalancer] enforce per-server move cap
The auto-rebalancer never enforced its per-server move cap. It budgeted
a whole batch at max_moves_per_server * num_tservers and checked
moves_per_tserver_, which is empty at selection time, so one batch could
pile many moves onto a single server (the most loaded one, which the cap
exists to protect).
Track per-server move counts within the batch and skip a move once
either endpoint reaches the cap.
Add MaxMovesPerServerEnforcedWithinOneBatch, which is red before this
change: a 6-move batch puts 3 moves on one server with the cap at 1.
Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Co-authored-by: Cursor <cu...@cursor.com>
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_rebalancer-test.cc
M src/kudu/master/auto_rebalancer.cc
3 files changed, 123 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/05/25005/5
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 2: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3091/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2026 14:42:37 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 2: Code-Review+1
(6 comments)
LGTM overall, a view nits.
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer-test.cc
File src/kudu/master/auto_rebalancer-test.cc:
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer-test.cc@1735
PS2, Line 1735: // The --auto_rebalancing_max_moves_per_server cap counts each server's moves
: // (as source or destination) separately, but GetMovesUsingRebalancingAlgo only
: // bounds the whole batch at max_moves_per_server * num_tservers, so one server
: // can source several moves in a single batch even with the cap set to 1.
Please update the description to be more explicit that the issue is now fixed.
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer-test.cc@1746
PS2, Line 1746: const int kNumOrigTservers = 3;
: const int kNumAdditionalTservers = 3;
: const int kNumTablets = 12;
nit: make these constexpr ?
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer-test.cc@1775
PS2, Line 1775: raw_info = rebalance::ClusterRawInfo();
It'd be cleaner to use a scope-local rebalance::ClusterRawInfo variable instead, and assign it to the outside 'raw_info' after ASSERT_FALSE().
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer-test.cc@1808
PS2, Line 1808: static_cast<int>(
nit: will it work without this extra cast?
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer-test.cc@1814
PS2, Line 1814: NO_PENDING_FATALS();
What is this for? In other words, where do we expect any pending fatals coming from at this point? Please add a comment if there is anything expected.
BTW, if using AI tools for generating the code, it's a good idea to reflect that in the commit description.
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer.cc
File src/kudu/master/auto_rebalancer.cc:
http://gerrit.cloudera.org:8080/#/c/25005/2/src/kudu/master/auto_rebalancer.cc@498
PS2, Line 498: rep_moves.back()
nit: SelectReplicaToMove() should return non-OK status when not a single move found, but it wouldn't hurt to add DCHECK(!rep_moves.empty()) to stipulate on this invariant
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Oct 2026 21:37:45 +0000
Gerrit-HasComments: Yes
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/3076/
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 02 Oct 2026 11:41:31 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
[auto_rebalancer] enforce per-server move cap
The auto-rebalancer never enforced its per-server move cap. It budgeted
a whole batch at max_moves_per_server * num_tservers and checked
moves_per_tserver_, which is empty at selection time, so one batch could
pile many moves onto a single server (the most loaded one, which the cap
exists to protect).
Track per-server move counts within the batch and skip a move once
either endpoint reaches the cap.
Add MaxMovesPerServerEnforcedWithinOneBatch, which is red before this
change: a 6-move batch puts 3 moves on one server with the cap at 1.
Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Co-authored-by: Cursor <cu...@cursor.com>
Reviewed-on: http://gerrit.cloudera.org:8080/25005
Reviewed-by: Alexey Serbin <al...@apache.org>
Tested-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_rebalancer-test.cc
M src/kudu/master/auto_rebalancer.cc
3 files changed, 123 insertions(+), 14 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 4: Code-Review+2
LGTM.
Please make sure the pre-commit test failures are unrelated to this changelist.
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 05:51:30 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 4: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3133/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Oct 2026 13:46:25 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 5: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/3142/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Oct 2026 14:46:06 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 5: Code-Review+2
> > Patch Set 4: Code-Review+2
> >
> > LGTM.
> >
> > Please make sure the pre-commit test failures are unrelated to
> this changelist.
>
> I noticed AddTserver in auto_leader_rebalancer-test.cc failed on
> the debug and release builds. It turned out the test is too slow
> now that the per-server cap is enforced, so I raised
> auto_rebalancing_max_moves_per_server to 5 in the test fixture.
> Without that change AddTserver failed locally, and with it, it
> passed 20 out of 20 repeated runs in a debug build.
Thanks for checking it out.
LGTM.
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 Oct 2026 22:15:20 +0000
Gerrit-HasComments: No
[kudu-CR] [auto rebalancer] enforce per-server move cap
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/25005 )
Change subject: [auto_rebalancer] enforce per-server move cap
......................................................................
Patch Set 5: Verified+1
DEBUG: unrelated test failures due to KUDU-3052 and KUDU-3660
--
To view, visit http://gerrit.cloudera.org:8080/25005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f6b9b200c24d215435329a37e953d75fe923c02
Gerrit-Change-Number: 25005
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 Oct 2026 22:18:29 +0000
Gerrit-HasComments: No