You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Yifan Zhang (Code Review)" <ge...@cloudera.org> on 2026/07/24 11:25:23 UTC
[kudu-CR] [tools/master] make move of a leader replica recoverable
Yifan Zhang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24630
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
[tools/master] make move of a leader replica recoverable
Moving a replica via 'kudu tablet change_config move_replica' or the
'kudu cluster rebalance' tool is client-driven: the source is marked with
the 'replace' attribute and, when it is the leader, the client asks it to
step down so it can be evicted (Raft cannot evict a leader). Both the
step-down and the wait-for-completion happen while the client keeps polling,
so if the client is interrupted (e.g. Ctrl+C, or an aborted/timed-out
rebalance run) after the config change but before the leader steps down, the
tablet is left over-replicated with a stuck, un-evictable leader.
Add two complementary recovery paths:
* Master-side self-healing: ShouldStepDownLeaderForReplacement() plus an
AsyncLeaderStepDownForReplacementTask scheduled from ProcessTabletReport()
once a leader marked with 'replace' has its replacement promoted to voter
(guarded by --catalog_manager_step_down_leader_for_replacement, on by
default, 3-4-3 only). It targets the captured leader and no-ops if that
replica is no longer the leader. This is driven by tablet state rather than
by whoever initiated the move, so it recovers an interrupted move_replica
as well as an interrupted 'cluster rebalance' run, with no client involved.
* CLI idempotency (move_replica only): IsReplicaMoveScheduled() lets
MoveReplica() detect an already-scheduled move and just wait for it to
finish, so re-running the command resumes instead of failing on the
over-replicated tablet.
Add a unit test for ShouldStepDownLeaderForReplacement() and an integration
test (TestMoveReplicaIdempotent) covering the interrupted-move resume.
Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Co-authored-by: Cursor <cu...@cursor.com>
---
M src/kudu/consensus/quorum_util-test.cc
M src/kudu/consensus/quorum_util.cc
M src/kudu/consensus/quorum_util.h
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/tool_action_tablet.cc
M src/kudu/tools/tool_replica_util.cc
M src/kudu/tools/tool_replica_util.h
8 files changed, 415 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24630/1
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 1
Gerrit-Owner: Yifan Zhang <ch...@163.com>
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Hello Gabriella Lotz, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24630
to look at the new patch set (#2).
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
[tools/master] make move of a leader replica recoverable
Moving a replica ('kudu tablet change_config move_replica' or 'kudu cluster
rebalance') is client-driven: the source is marked 'replace' and, if it is
the leader, the client asks it to step down so it can be evicted (Raft can't
evict a leader). If the client is interrupted (Ctrl+C, or an aborted/timed-out
rebalance) after the config change but before the leader steps down, the
tablet is left over-replicated with a stuck, un-evictable leader.
Add two recovery paths:
* Master-side self-healing: once a leader marked 'replace' has its replacement
promoted to voter, ProcessTabletReport() schedules an
AsyncLeaderStepDownForReplacementTask to step it down so it can be evicted
(see ShouldStepDownLeaderForReplacement(); guarded by
--catalog_manager_step_down_leader_for_replacement, on by default, 3-4-3
only). Driven by tablet state, so it recovers an interrupted move_replica or
'cluster rebalance' with no client involved.
* CLI idempotency (move_replica): IsReplicaMoveScheduled() lets MoveReplica()
detect an already-scheduled move and wait for it, so re-running resumes
instead of failing on the over-replicated tablet.
Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Co-authored-by: Cursor <cu...@cursor.com>
---
M src/kudu/consensus/quorum_util-test.cc
M src/kudu/consensus/quorum_util.cc
M src/kudu/consensus/quorum_util.h
M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/tool_action_tablet.cc
M src/kudu/tools/tool_replica_util.cc
M src/kudu/tools/tool_replica_util.h
9 files changed, 520 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24630/2
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 2
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 3:
Build Aborted
http://jenkins.kudu.apache.org/job/pre_commit/2823/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 3
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 00:06:39 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 2:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2791/
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 2
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 28 Jul 2026 12:38:20 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 4: Verified+1
DEBUG: unrelated test failure in webserver-crawl-itest due to CI pipeline issue
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 4
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 22:42:56 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 4: Code-Review+2
Thank you very much for making moving a leader replica more robust!
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 4
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 22:59:56 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 2: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2791/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 2
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 14:45:12 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2782/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 1
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Jul 2026 12:22:43 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc
File src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc:
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc@1340
PS2, Line 1340: ASSERT_OK(GetConsensusState(observer, tablet_id_, kTimeout,
> nit: the disabled-case relies on a fixed SleepFor(5s) to assert the master
Good point.
Following the same pattern used in raft_consensus-itest.cc, I've wrapped the sleep and the "never evicts" assertions in AllowSlowTests().
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc@5517
PS2, Line 5517: bool AsyncLeaderStepDownForReplacementTask::SendRequest(int attempt) {
> q: If the 'replace'-marked leader steps down but is not yet evicted, nothin
Yes, I think that's intended and safe and that's not a real ping-pong.
1. While it's leader, a replace-marked replica can't self-elect — BecomeLeaderUnlocked() calls DisableFailureDetector() .
2. The step-down is GRACEFUL, i.e. TransferLeadership() hands off to a caught-up voter; on granting its vote the ex-leader snoozes its own failure detector (RequestVoteRespondVoteGranted, raft_consensus.cc:2454-2466), so it won't immediately re-run and the new leader gets a window to evict it.
3. If the transfer never completes, it simply stays leader (never stepped down) and the master just retries on the next report — no ping-pong there either.
So the worst case is a bounded number of idempotent step-down RPCs.
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc@5900
PS2, Line 5900: rpcs.emplace_back(new AsyncLeaderStepDownForReplacementTask(
> q: Unlike the AsyncEvictReplicaTask / AsyncAddReplicaTask branches, a leade
Good catch. But the churn is already bounded: before dispatch, ProcessTabletReport() dedups queued RPCs by (tablet_id, description) via ContainsTask() (catalog_manager.cc:6008), and a task stays in pending_tasks_ for its whole retry lifetime. The step-down task's description includes the tablet id and leader uuid (stable during the in-flight transfer), so repeated reports just hit "Not sending duplicate request". So an extra gate would be largely redundant.
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc
File src/kudu/tools/tool_replica_util.cc:
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc@638
PS2, Line 638:
> nit: IsReplicaMoveScheduled() runs GetTabletLeader()/GetConsensusState() wi
You're right.
We considered adding a bounded retry around the check, but decided against it: it just introduces another hard-coded 5s magic number (on top of the existing ksck one) to paper over a rare, transient window.
Since move_replica is a manual command and is now idempotent, letting the operator just re-run it is the cleaner option.
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc@668
PS2, Line 668: (!from_ts_uuid_in_config || from_ts_uuid_has_replace);
> q: In the 3-4-3 branch, `scheduled` becomes true when `to` is present and `
Good observation.
But validating that from was a genuine replica doesn't really work here, because two different situations produce the same config state, and cstate carries no marker to tell them apart:
1. The legitimate idempotent case: a move that already completed — to was added and promoted, and from was evicted, so it's naturally gone from the config. This is exactly the state the check is meant to recognize so a re-run resumes and reports completion.
2. The bad-input case you describe: from was never a replica, and to happens to already be one.
Given that it needs compounded invalid input (bogus from + to already a voter) and the outcome is a harmless misleading "success" with no config change, I'd treat it as out of scope.
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 3
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 11:15:48 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 2:
(5 comments)
Thanks for working on this!
Overall looks good, just some observations.
Thanks!
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc
File src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc:
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc@1340
PS2, Line 1340: EXCLUDE_HEALTH_REPORT, &cs));
nit: the disabled-case relies on a fixed SleepFor(5s) to assert the master never evicts. That is inherently the right shape for a negative test, but it fixes a 5s floor on this parameter instance. Fine to leave; just flagging the cost.
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc@5517
PS2, Line 5517: bool AsyncLeaderStepDownForReplacementTask::SendRequest(int attempt) {
q: If the 'replace'-marked leader steps down but is not yet evicted, nothing prevents it from winning a subsequent election (the 'replace' attr does not exclude it from candidacy). If it re-becomes leader before eviction commits, this task path re-triggers. Presumably eventual eviction wins the race and this is the same self-healing loop the client path also relies on -- just confirming that's the intended/acceptable behavior rather than a potential ping-pong.
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/master/catalog_manager.cc@5900
PS2, Line 5900: rpcs.emplace_back(new AsyncLeaderStepDownForReplacementTask(
q: Unlike the AsyncEvictReplicaTask / AsyncAddReplicaTask branches, a leader step-down does not create a pending config, so this branch is not suppressed by the enclosing `!cstate.has_pending_config()` guard. That means while the graceful leadership transfer is in flight the leader still reports itself as leader marked 'replace' on every heartbeat, and each report schedules a fresh AsyncLeaderStepDownForReplacementTask. It looks self-limiting (redundant tasks get ServiceUnavailable "transfer already in progress" and retry, then MarkComplete once SendRequest/HandleResponse observe the leader changed), but at low --heartbeat_interval_ms this can spawn a burst of tasks and repeated GRACEFUL step-down RPCs. Is it worth gating on an already-in-flight step-down for this tablet (or otherwise deduping) to avoid the churn?
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc
File src/kudu/tools/tool_replica_util.cc:
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc@638
PS2, Line 638:
nit: IsReplicaMoveScheduled() runs GetTabletLeader()/GetConsensusState() with no retry, and it is now the first thing MoveReplica() does. If the tablet is transiently leaderless -- e.g. exactly during the recovery step-down this feature detects -- the command fails outright instead of retrying (the old WaitForCleanKsck path retried for 5s). Minor, since move_replica is a manual command, but worth a thought.
http://gerrit.cloudera.org:8080/#/c/24630/2/src/kudu/tools/tool_replica_util.cc@668
PS2, Line 668: (!from_ts_uuid_in_config || from_ts_uuid_has_replace);
q: In the 3-4-3 branch, `scheduled` becomes true when `to` is present and `from` is absent (!from_ts_uuid_in_config). If a caller passes a from_ts_uuid that is not actually a current replica while `to` coincidentally is, this reports "already scheduled" and MoveReplica() skips scheduling and drops straight into WaitForMoveToComplete -- whereas the previous flow would have surfaced an error via ScheduleReplicaMove(). Is validating that `from` was a genuine replica worth it here, or is that input out of scope?
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 2
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 13:20:13 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 2
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 15:54:53 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2782/
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 1
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 24 Jul 2026 11:25:31 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 4
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Gabriella Lotz (Code Review)" <ge...@cloudera.org>.
Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 1:
(2 comments)
Thank you for working on this!
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/master/catalog_manager.cc@5492
PS1, Line 5492: const ConsensusStatePB cstate_;
nit: it looks like only leader_uuid_ is read after construction, so the full cstate_ copy seems unused. Could drop it, though I see the other tasks keep cstate_ around too, so feel free to leave it for consistency.
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/tools/kudu-admin-test.cc
File src/kudu/tools/kudu-admin-test.cc:
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/tools/kudu-admin-test.cc@499
PS1, Line 499: TEST_F(AdminCliTest, TestMoveReplicaIdempotent) {
Would it be feasible to add a case that leaves a leader marked replace with its replacement promoted, then asserts the master steps it down and evicts it with no client involved? I think that'd exercise AsyncLeaderStepDownForReplacementTask directly.
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 1
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 27 Jul 2026 11:31:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 4: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2888/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 4
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 22:15:16 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 3:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2823/
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 3
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 11:15:10 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/master/catalog_manager.cc@5492
PS1, Line 5492: const string leader_uuid_;
> nit: it looks like only leader_uuid_ is read after construction, so the ful
Done
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/tools/kudu-admin-test.cc
File src/kudu/tools/kudu-admin-test.cc:
http://gerrit.cloudera.org:8080/#/c/24630/1/src/kudu/tools/kudu-admin-test.cc@499
PS1, Line 499: TEST_F(AdminCliTest, TestMoveReplicaIdempotent) {
> Would it be feasible to add a case that leaves a leader marked replace with
Good idea, thanks! Done in PS2.
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 2
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 12:44:44 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.
Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 3: Code-Review+2
Thanks for the details, LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 3
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 11:53:45 +0000
Gerrit-HasComments: No
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Gabriella Lotz, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24630
to look at the new patch set (#3).
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
[tools/master] make move of a leader replica recoverable
Moving a replica ('kudu tablet change_config move_replica' or 'kudu cluster
rebalance') is client-driven: the source is marked 'replace' and, if it is
the leader, the client asks it to step down so it can be evicted (Raft can't
evict a leader). If the client is interrupted (Ctrl+C, or an aborted/timed-out
rebalance) after the config change but before the leader steps down, the
tablet is left over-replicated with a stuck, un-evictable leader.
Add two recovery paths:
* Master-side self-healing: once a leader marked 'replace' has its replacement
promoted to voter, ProcessTabletReport() schedules an
AsyncLeaderStepDownForReplacementTask to step it down so it can be evicted
(see ShouldStepDownLeaderForReplacement(); guarded by
--catalog_manager_step_down_leader_for_replacement, on by default, 3-4-3
only). Driven by tablet state, so it recovers an interrupted move_replica or
'cluster rebalance' with no client involved.
* CLI idempotency (move_replica): IsReplicaMoveScheduled() lets MoveReplica()
detect an already-scheduled move and wait for it, so re-running resumes
instead of failing on the over-replicated tablet.
Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Co-authored-by: Cursor <cu...@cursor.com>
---
M src/kudu/consensus/quorum_util-test.cc
M src/kudu/consensus/quorum_util.cc
M src/kudu/consensus/quorum_util.h
M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/tool_action_tablet.cc
M src/kudu/tools/tool_replica_util.cc
M src/kudu/tools/tool_replica_util.h
9 files changed, 522 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24630/3
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 3
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
[tools/master] make move of a leader replica recoverable
Moving a replica ('kudu tablet change_config move_replica' or 'kudu cluster
rebalance') is client-driven: the source is marked 'replace' and, if it is
the leader, the client asks it to step down so it can be evicted (Raft can't
evict a leader). If the client is interrupted (Ctrl+C, or an aborted/timed-out
rebalance) after the config change but before the leader steps down, the
tablet is left over-replicated with a stuck, un-evictable leader.
Add two recovery paths:
* Master-side self-healing: once a leader marked 'replace' has its replacement
promoted to voter, ProcessTabletReport() schedules an
AsyncLeaderStepDownForReplacementTask to step it down so it can be evicted
(see ShouldStepDownLeaderForReplacement(); guarded by
--catalog_manager_step_down_leader_for_replacement, on by default, 3-4-3
only). Driven by tablet state, so it recovers an interrupted move_replica or
'cluster rebalance' with no client involved.
* CLI idempotency (move_replica): IsReplicaMoveScheduled() lets MoveReplica()
detect an already-scheduled move and wait for it, so re-running resumes
instead of failing on the over-replicated tablet.
Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Co-authored-by: Cursor <cu...@cursor.com>
Reviewed-on: http://gerrit.cloudera.org:8080/24630
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/consensus/quorum_util-test.cc
M src/kudu/consensus/quorum_util.cc
M src/kudu/consensus/quorum_util.h
M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/tool_action_tablet.cc
M src/kudu/tools/tool_replica_util.cc
M src/kudu/tools/tool_replica_util.h
9 files changed, 522 insertions(+), 9 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 5
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
[kudu-CR] [tools/master] make move of a leader replica recoverable
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
Patch Set 4:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2888/
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 4
Gerrit-Owner: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Gabriella Lotz <lo...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 21:24:05 +0000
Gerrit-HasComments: No