You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Gabriella Lotz (Code Review)" <ge...@cloudera.org> on 2026/09/25 14:03:46 UTC

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup, not an interval later

Gabriella Lotz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24946


Change subject: KUDU-3733 run auto-leader-rebalancer at startup, not an interval later
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup, not an interval later

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 63 insertions(+), 8 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/24946/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 4: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/3089/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 14:47:18 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 6:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc@1001
PS5, Line 1001: poll_interval;
> nit: could this be computed just once outside of the while() cycle?  Also, 
Done, I left it as a plain const since MonoDelta::FromSeconds() isn't constexpr.


http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc@1008
PS5, Line 1008:     if (shutdown_.WaitFor(wait)) {
              :       break;
              :     }
> Does this mean it now will wait for 5 seconds when shutting down a kudu-mas
I don't think so. WaitFor() is on the shutdown_ latch, so Shutdown() wakes the thread right away rather than waiting out the 5 seconds. The poll interval is only the maximum sleep between checks.



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Oct 2026 13:34:16 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 3: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/3070/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 15:17:54 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has removed a vote on this change.

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 5:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/3131/


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Oct 2026 11:58:13 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24946

to look at the new patch set (#4).

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

While here, clamp the interval to a second. The flag has no validator
and is runtime settable, so 0 is reachable, and at 0 the loop never
sleeps and spins on RunLeaderRebalancer() (~14k passes in 3 seconds,
measured). That spin predates this patch and isn't made worse by it,
but the clamp is cheap, and reading the interval once for both the
poll and the due check beats reading the flag in two places.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop, and ZeroIntervalDoesNotSpin, which fails
without the clamp.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 103 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/24946/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Alexey Serbin, Zoltan Martonka, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24946

to look at the new patch set (#7).

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

While here, clamp the interval to a second. The flag has no validator
and is runtime settable, so 0 is reachable, and at 0 the loop never
sleeps and spins on RunLeaderRebalancer() (~14k passes in 3 seconds,
measured). That spin predates this patch and isn't made worse by it,
but the clamp is cheap, and reading the interval once for both the
poll and the due check beats reading the flag in two places.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop, and ZeroIntervalDoesNotSpin, which fails
without the clamp.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Co-authored-by: Cursor <cu...@cursor.com>
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 104 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/24946/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 5: Verified+1

Build Successful 

http://jenkins.kudu.apache.org/job/pre_commit/3131/ : SUCCESS


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Oct 2026 13:00:23 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 1: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/3048/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 25 Sep 2026 14:40:23 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24946/2/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/2/src/kudu/master/auto_leader_rebalancer.cc@997
PS2, Line 997:       if (ran) {
> The flag has no validator, so --auto_leader_rebalancing_interval_seconds=0 
Thanks for pointing this out! I went with the clamp. The interval is read once per iteration now and floored at a second, so one value drives both the due check and the poll, which also folds away the double read you noticed.

I preferred that over a validator because the flag is runtime-tagged, so a validator would start rejecting set_flag calls and startup configs that are accepted today.



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 14:22:38 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 6: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/3195/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Oct 2026 14:26:50 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 6:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/3195/


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Oct 2026 13:33:50 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 6: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc@1008
PS5, Line 1008:     if (shutdown_.WaitFor(wait)) {
              :       break;
              :     }
> I don't think so. WaitFor() is on the shutdown_ latch, so Shutdown() wakes 
Right, indeed -- sorry for the noise.  Should be good as-is.


http://gerrit.cloudera.org:8080/#/c/24946/6/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/6/src/kudu/master/auto_leader_rebalancer.cc@969
PS6, Line 969:   const MonoDelta poll_interval = MonoDelta::FromSeconds(5);
nit: this could be 'static', but I guess it's totally OK to have as-is now; sometime, once switching MonoDelta to constexpr, we can update all the corresponding calls sites accordingly



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Oct 2026 17:48:52 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 7: Verified+1 Code-Review+2

unrelated test failures:
tsan: org.apache.kudu.spark.kudu.DefaultSourceTest, org.apache.kudu.backup.TestKuduBackup


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Oct 2026 15:30:18 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup, not an interval later

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup, not an interval later
......................................................................


Patch Set 1:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/3048/


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 25 Sep 2026 14:03:55 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24946/2/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/2/src/kudu/master/auto_leader_rebalancer.cc@997
PS2, Line 997:   } while (!shutdown_.WaitFor(MonoDelta::FromSeconds(
The flag has no validator, so --auto_leader_rebalancing_interval_seconds=0 is settable (it's runtime-tagged too). With 0, std::min(kPollIntervalSecs, 0) makes the tail WaitFor(FromSeconds(0)) return immediately, and pass_is_due (now - last_run_time >= FromSeconds(0)) is always true — so the loop hammers RunLeaderRebalancer() in a tight spin. This is pre-existing (the old loop also busy-looped on 0), so not introduced here, but this patch now reads the interval in two spots and the framing is "interval isn't how long to sleep." Worth deciding: clamp the poll to a floor (e.g. std::min(kPollIntervalSecs, std::max(1u, interval))) / add a GROUP_FLAG_VALIDATOR, or explicitly call it out of scope? A one-line comment noting 0 is degenerate would also be fine.



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Mon, 28 Sep 2026 13:18:37 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

Just one more question. Otherwise looks good.

http://gerrit.cloudera.org:8080/#/c/24946/3/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/3/src/kudu/master/auto_leader_rebalancer.cc@1001
PS3, Line 1001:     if (shutdown_.WaitFor(std::min(MonoDelta::FromSeconds(kPollIntervalSecs), interval))) {
The fine poll is only needed while we're waiting to acquire (or re-acquire) leadership. Once a pass has run and we're the steady-state leader, we already know the next pass is due at last_run_time + interval, yet we still wake every kPollIntervalSecs just to find pass_is_due == false and sleep again (~720 wakeups/hour at the default). Could the wait be min(poll, time_until_due) instead -- poll finely only when !pass_is_due for leadership reasons, and otherwise sleep until the pass is actually due? Per-wake cost is trivial so this isn't blocking; mostly asking whether the uniform poll is the intended shape or just the simplest one.



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 16:27:02 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24946

to look at the new patch set (#2).

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 63 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/24946/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24946

to look at the new patch set (#3).

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

While here, clamp the interval to a second. The flag has no validator
and is runtime settable, so 0 is reachable, and at 0 the loop never
sleeps and spins on RunLeaderRebalancer() (~14k passes in 3 seconds,
measured). That spin predates this patch and isn't made worse by it,
but the clamp is cheap, and reading the interval once for both the
poll and the due check beats reading the flag in two places.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop, and ZeroIntervalDoesNotSpin, which fails
without the clamp.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 96 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/24946/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Sat, 31 Oct 2026 00:45:33 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 4:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/3089/


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 13:45:57 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Zoltan Martonka (Code Review)" <ge...@cloudera.org>.
Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 5: Code-Review+1

LGTM


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Oct 2026 07:41:46 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

While here, clamp the interval to a second. The flag has no validator
and is runtime settable, so 0 is reachable, and at 0 the loop never
sleeps and spins on RunLeaderRebalancer() (~14k passes in 3 seconds,
measured). That spin predates this patch and isn't made worse by it,
but the clamp is cheap, and reading the interval once for both the
poll and the due check beats reading the flag in two places.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop, and ZeroIntervalDoesNotSpin, which fails
without the clamp.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Co-authored-by: Cursor <cu...@cursor.com>
Reviewed-on: http://gerrit.cloudera.org:8080/24946
Reviewed-by: Marton Greber <gr...@gmail.com>
Tested-by: Marton Greber <gr...@gmail.com>
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 104 insertions(+), 10 deletions(-)

Approvals:
  Marton Greber: Looks good to me, approved; Verified
  Alexey Serbin: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 8
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 7: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/3202/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Oct 2026 15:13:54 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 6:

In tsan there is data race reported in LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval

Otherwise looks good to me!


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Oct 2026 11:46:04 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 3:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/3070/


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 14:22:37 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc@1001
PS5, Line 1001: MonoDelta::FromSeconds(kPollIntervalSecs);
nit: could this be computed just once outside of the while() cycle?  Also, it might be even constexpr if MonoDelta::FromSeconds() would allow for that (the latter might might be a separate theme on its own, I guess).


http://gerrit.cloudera.org:8080/#/c/24946/5/src/kudu/master/auto_leader_rebalancer.cc@1008
PS5, Line 1008:     if (shutdown_.WaitFor(wait)) {
              :       break;
              :     }
Does this mean it now will wait for 5 seconds when shutting down a kudu-master process that doesn't have the leader role for the system tablet?  If so, is there a way to avoid that?



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Oct 2026 06:37:07 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 7:

(1 comment)

> Patch Set 6:
> 
> In tsan there is data race reported in LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval
> 
> Otherwise looks good to me!

Thanks for catching this! It was in the new tests. They changed --auto_leader_rebalancing_interval_seconds and --auto_leader_rebalancing_enabled after the cluster had already started, while the loop was re-reading them on every poll. The old loop read the interval only once before its first long sleep, so it never hit this. I've changed both tests to set the flags before starting the cluster.

http://gerrit.cloudera.org:8080/#/c/24946/6/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/6/src/kudu/master/auto_leader_rebalancer.cc@969
PS6, Line 969:   static const MonoDelta poll_interval = MonoDelta::FromSeconds(5);
> nit: this could be 'static', but I guess it's totally OK to have as-is now;
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Oct 2026 14:13:28 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 7:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/3202/


-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 7
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Oct 2026 14:01:10 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Alexey Serbin, Zoltan Martonka, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24946

to look at the new patch set (#6).

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................

KUDU-3733 run auto-leader-rebalancer at startup

RunLoop() slept for --auto_leader_rebalancing_interval_seconds before
its first pass, leaving the auto-leader-rebalancer idle for up to an
hour after master startup or failover.

Running a pass immediately isn't enough on its own: the thread is
created while the catalog manager is still initializing, so that pass
no-ops on the leadership check and the loop sleeps out the interval
anyway. The interval says how often to rebalance, not how long to
sleep, so separate the two. The loop now polls every 5 seconds and
decides whether a pass is due from when the last one ran; only a pass
that got past the leadership check advances that timestamp.

While here, clamp the interval to a second. The flag has no validator
and is runtime settable, so 0 is reachable, and at 0 the loop never
sleeps and spins on RunLeaderRebalancer() (~14k passes in 3 seconds,
measured). That spin predates this patch and isn't made worse by it,
but the clamp is cheap, and reading the interval once for both the
poll and the due check beats reading the flag in two places.

Adds LeaderRebalancerTest.FirstPassDoesNotWaitOutFullInterval, which
fails against the old loop, and ZeroIntervalDoesNotSpin, which fails
without the clamp.

Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
---
M src/kudu/master/auto_leader_rebalancer-test.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_leader_rebalancer.h
3 files changed, 103 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/24946/6
-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 6
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] KUDU-3733 run auto-leader-rebalancer at startup

Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24946 )

Change subject: KUDU-3733 run auto-leader-rebalancer at startup
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24946/3/src/kudu/master/auto_leader_rebalancer.cc
File src/kudu/master/auto_leader_rebalancer.cc:

http://gerrit.cloudera.org:8080/#/c/24946/3/src/kudu/master/auto_leader_rebalancer.cc@1001
PS3, Line 1001:     MonoDelta wait = MonoDelta::FromSeconds(kPollIntervalSecs);
> The fine poll is only needed while we're waiting to acquire (or re-acquire)
Done, good point. Switched to sleeping until the next pass is due and kept the fine poll only while acquiring leadership.



-- 
To view, visit http://gerrit.cloudera.org:8080/24946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3c1b2a5622578c14a414fc79e4c527f9f7efbbce
Gerrit-Change-Number: 24946
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Oct 2026 13:46:00 +0000
Gerrit-HasComments: Yes