You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Gabriella Lotz (Code Review)" <ge...@cloudera.org> on 2026/07/16 12:42:58 UTC
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Gabriella Lotz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24591
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
KUDU-3789 Deflake auto_leader_rebalancer-test
Under TSAN this test suite was very flaky (about 488 of 1000 runs
failed at one baseline). The causes were all test-side timing
assumptions on a cluster whose leadership keeps moving, not product
bugs.
Fixed retry loops (run N rounds, then assert balanced) raced the
cluster's ongoing leadership moves. Replaced them with a
RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over
a TSAN-aware time budget via AssertEventually.
AddTserver and RestartTserver used 59 tablets. That many raft groups
move leadership around through natural elections and keep the cluster
from ever reaching an exactly balanced snapshot. Reduced to 24
(divisible by the tserver count for a clean balance target). Note that
stopping the replica rebalancer to remove that movement makes things
worse: on a static cluster the leader rebalancer gets stuck moving the
same leaders back and forth and never balances, so the replica
rebalancer is left running.
Placement-sensitive tests (MultiTableLeaderBalance,
FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by
extra leader elections when TSAN CPU starvation exceeded the default
leader failure timeout. Raised
leader_failure_max_missed_heartbeat_periods from its default of 3.0 to
15.0 for those tests (none shut a tserver down, so slower failover does
no harm), and reset it to the default in TearDown so the failover-based
tests are unaffected under test reordering.
Relaxed the ignored soft-deleted table check in
FilterSoftDeletedTableTest to "not fully balanced" rather than requiring
every leader count to differ from 3, which natural elections could trip.
Rewrote TestMaintenanceMode to arrange the imbalance directly with
MakeLeaderDistribution and assert the rebalancer never places a leader
on the maintenance-mode tserver, instead of restarting the tserver and
racing on election timing.
Test-only. No production behavior change.
Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
---
M src/kudu/master/auto_leader_rebalancer-test.cc
1 file changed, 140 insertions(+), 169 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/91/24591/1
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24591
to look at the new patch set (#4).
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
KUDU-3789 Deflake auto_leader_rebalancer-test
Under TSAN this test suite was very flaky (about 488 of 1000 runs
failed at one baseline). The causes were all test-side timing
assumptions on a cluster whose leadership keeps moving, not product
bugs.
Fixed retry loops (run N rounds, then assert balanced) raced the
cluster's ongoing leadership moves. Replaced them with a
RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over
a TSAN-aware time budget via AssertEventually.
AddTserver and RestartTserver used 59 tablets. That many raft groups
move leadership around through natural elections and keep the cluster
from ever reaching an exactly balanced snapshot. Note that stopping
the replica rebalancer to remove that movement makes things worse: on
a static cluster the leader rebalancer gets stuck moving the same
leaders back and forth and never balances, so the replica rebalancer
is left running.
Placement-sensitive tests (MultiTableLeaderBalance,
FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by
extra leader elections when TSAN CPU starvation exceeded the default
leader failure timeout.
Relaxed the ignored soft-deleted table check in
FilterSoftDeletedTableTest to "not fully balanced" rather than requiring
every leader count to differ from 3, which natural elections could trip.
Rewrote TestMaintenanceMode to arrange the imbalance directly with
MakeLeaderDistribution and assert the rebalancer never places a leader
on the maintenance-mode tserver, instead of restarting the tserver and
racing on election timing.
Test-only. No production behavior change.
Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
---
M src/kudu/master/auto_leader_rebalancer-test.cc
1 file changed, 158 insertions(+), 167 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/91/24591/4
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 4: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2871/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Aug 2026 11:11:06 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 1:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2753/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 17 Jul 2026 00:43:12 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24591/2/src/kudu/master/auto_leader_rebalancer-test.cc
File src/kudu/master/auto_leader_rebalancer-test.cc:
http://gerrit.cloudera.org:8080/#/c/24591/2/src/kudu/master/auto_leader_rebalancer-test.cc@118
PS2, Line 118: // Restore the default in case a test bumped it (gflags are process global,
> q: This manual reset (and the whole "reset in TearDown so failover-based te
Ah, you're right, thanks. I've dropped the line and the commit message note.
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 13:27:20 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24591/3/src/kudu/master/auto_leader_rebalancer-test.cc
File src/kudu/master/auto_leader_rebalancer-test.cc:
http://gerrit.cloudera.org:8080/#/c/24591/3/src/kudu/master/auto_leader_rebalancer-test.cc@505
PS3, Line 505: // Keep the tablet count modest so natural-election churn under TSAN doesn't
: // perpetually prevent an exactly balanced leader snapshot. Divisible by the
: // tserver count for a clean balance target.
: const int kNumTablets = 24;
I'm curious why the original choice was 59. Probably, there was an idea to see how the rebalancer behaves when the balance balance isn't 'clean', and maybe that's a good idea to keep it at least for non-TSAN builds.
If it makes sense, consider different settings for kNumTablets based on whether it's a sanitized build or not. It's quite a common theme: search for THREAD_SANITIZER and ADDRESS_SANITIZER in alter_table-test.cc and other tests.
http://gerrit.cloudera.org:8080/#/c/24591/3/src/kudu/master/auto_leader_rebalancer-test.cc@1019
PS3, Line 1019: FLAGS_leader_failure_max_missed_heartbeat_periods = 15.0;
ditto: consider introducing this customization only for THREAD_SANITIZER case -- perhaps, it makes sense to test how auto-leader rebalancing behaves with default settings in other type of builds where the rate of spurious re-elections is not that high, but a few re-elections can happen during the course of leader rebalancing.
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 00:20:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 00:20:26 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2753/
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 1
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 16 Jul 2026 12:43:09 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 4: Verified+1
unrelated failures of known flaky tests:
* ASAN: cfile-test failing because of disk space issues (suspecting running http://gerrit.cloudera.org:8080/24412 to be the root cause)
* TSAN: org.apache.kudu.client.TestSecurity, org.apache.kudu.backup.TestKuduBackup
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 02:03:43 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 3:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2824/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 01:45:21 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
KUDU-3789 Deflake auto_leader_rebalancer-test
Under TSAN this test suite was very flaky (about 488 of 1000 runs
failed at one baseline). The causes were all test-side timing
assumptions on a cluster whose leadership keeps moving, not product
bugs.
Fixed retry loops (run N rounds, then assert balanced) raced the
cluster's ongoing leadership moves. Replaced them with a
RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over
a TSAN-aware time budget via AssertEventually.
AddTserver and RestartTserver used 59 tablets. That many raft groups
move leadership around through natural elections and keep the cluster
from ever reaching an exactly balanced snapshot. Note that stopping
the replica rebalancer to remove that movement makes things worse: on
a static cluster the leader rebalancer gets stuck moving the same
leaders back and forth and never balances, so the replica rebalancer
is left running.
Placement-sensitive tests (MultiTableLeaderBalance,
FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by
extra leader elections when TSAN CPU starvation exceeded the default
leader failure timeout.
Relaxed the ignored soft-deleted table check in
FilterSoftDeletedTableTest to "not fully balanced" rather than requiring
every leader count to differ from 3, which natural elections could trip.
Rewrote TestMaintenanceMode to arrange the imbalance directly with
MakeLeaderDistribution and assert the rebalancer never places a leader
on the maintenance-mode tserver, instead of restarting the tserver and
racing on election timing.
Test-only. No production behavior change.
Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Reviewed-on: http://gerrit.cloudera.org:8080/24591
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/master/auto_leader_rebalancer-test.cc
1 file changed, 158 insertions(+), 167 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 5
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2771/
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 21 Jul 2026 11:40:27 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24591/3/src/kudu/master/auto_leader_rebalancer-test.cc
File src/kudu/master/auto_leader_rebalancer-test.cc:
http://gerrit.cloudera.org:8080/#/c/24591/3/src/kudu/master/auto_leader_rebalancer-test.cc@505
PS3, Line 505: // Create a cluster, and create a table,
: // whether tablets is balanced, which decided by creating table process.
: // Bring up another tserver, the table is not balanced and leaders is also
: // not balanced. We verify th
> I'm curious why the original choice was 59. Probably, there was an idea to
Done
http://gerrit.cloudera.org:8080/#/c/24591/3/src/kudu/master/auto_leader_rebalancer-test.cc@1019
PS3, Line 1019: // A few RF=1 tables: the sole replica is the leader and cannot be moved.
> ditto: consider introducing this customization only for THREAD_SANITIZER ca
Done
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Aug 2026 10:17:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 3: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2830/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 06:32:38 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24591/2/src/kudu/master/auto_leader_rebalancer-test.cc
File src/kudu/master/auto_leader_rebalancer-test.cc:
http://gerrit.cloudera.org:8080/#/c/24591/2/src/kudu/master/auto_leader_rebalancer-test.cc@118
PS2, Line 118: // Restore the default in case a test bumped it (gflags are process global,
q: This manual reset (and the whole "reset in TearDown so failover-based tests are unaffected under reordering" motivation in the commit message) looks redundant. KuduTest holds a google::FlagSaver constructed in its ctor and reset in its dtor (test_util.cc:124,172), so every gflag is already restored to its pre-test value after each TEST_F. Is there a case FlagSaver doesn't cover here? If not, this line could be dropped. If we keep it as belt-and-suspenders, note that hardcoding 3.0 duplicates the DEFINE_double default in raft_consensus.cc:81 and would silently go stale if that default ever changes -- capturing/restoring via google::FlagSaver locally (or just relying on the base-class one) avoids the magic number.
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 12:54:40 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 4:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2871/
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 11 Aug 2026 10:17:01 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24591
to look at the new patch set (#3).
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
KUDU-3789 Deflake auto_leader_rebalancer-test
Under TSAN this test suite was very flaky (about 488 of 1000 runs
failed at one baseline). The causes were all test-side timing
assumptions on a cluster whose leadership keeps moving, not product
bugs.
Fixed retry loops (run N rounds, then assert balanced) raced the
cluster's ongoing leadership moves. Replaced them with a
RunUntilLeaderBalanced() helper that retries CheckLeaderBalance() over
a TSAN-aware time budget via AssertEventually.
AddTserver and RestartTserver used 59 tablets. That many raft groups
move leadership around through natural elections and keep the cluster
from ever reaching an exactly balanced snapshot. Reduced to 24
(divisible by the tserver count for a clean balance target). Note that
stopping the replica rebalancer to remove that movement makes things
worse: on a static cluster the leader rebalancer gets stuck moving the
same leaders back and forth and never balances, so the replica
rebalancer is left running.
Placement-sensitive tests (MultiTableLeaderBalance,
FilterSoftDeletedTableTest, TestMaintenanceMode) were disrupted by
extra leader elections when TSAN CPU starvation exceeded the default
leader failure timeout. Raised
leader_failure_max_missed_heartbeat_periods from its default of 3.0 to
15.0 for those tests (none shut a tserver down, so slower failover does
no harm).
Relaxed the ignored soft-deleted table check in
FilterSoftDeletedTableTest to "not fully balanced" rather than requiring
every leader count to differ from 3, which natural elections could trip.
Rewrote TestMaintenanceMode to arrange the imbalance directly with
MakeLeaderDistribution and assert the rebalancer never places a leader
on the maintenance-mode tserver, instead of restarting the tserver and
racing on election timing.
Test-only. No production behavior change.
Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
---
M src/kudu/master/auto_leader_rebalancer-test.cc
1 file changed, 137 insertions(+), 169 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/91/24591/3
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 2: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2771/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 2
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 21 Jul 2026 13:21:02 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 4: Code-Review+2
Thank you for addressing the flakiness in the newly introduced test scenarios!
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 4
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 02:04:55 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 3:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2824/
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 13:27:21 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3789 Deflake auto leader rebalancer-test
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24591 )
Change subject: KUDU-3789 Deflake auto_leader_rebalancer-test
......................................................................
Patch Set 3:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2830/
--
To view, visit http://gerrit.cloudera.org:8080/24591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I00224fddfead0fb462e793f085d935b597792aaa
Gerrit-Change-Number: 24591
Gerrit-PatchSet: 3
Gerrit-Owner: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 04:13:31 +0000
Gerrit-HasComments: No