You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2016/10/12 06:49:45 UTC

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections 

Hello David Ribeiro Alves, Mike Percy,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/4694

to review the following change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................

WIP: KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

WIP because this needs tests, etc.

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/integration-tests/raft_consensus-itest.cc
8 files changed, 152 insertions(+), 72 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/4694/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Mike Percy, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/4694

to look at the new patch set (#2).

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................

WIP: KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

WIP because this needs tests, etc.

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/tserver/tablet_service.cc
10 files changed, 211 insertions(+), 99 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/4694/2
-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 1:

One more TODO item here is that LeaderElectionExpBackoff needs to use a different count for figuring out how many leader elections have failed, since we are no longer bumping through terms during failed leader elections in many cases.

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Mike Percy, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/4694

to look at the new patch set (#4).

Change subject: KUDU-1365. Add leader pre-elections
......................................................................

KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

Cluster tests verify that this substantially reduces the election storms
after a node has a temporarily slow disk or otherwise freezes.

A new experimental flag 'raft_enable_pre_election' is introduced which
defaults the feature to on, but provides a safety valve to disable this
in case we find some bug after release.

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/tserver/tablet_service.cc
10 files changed, 262 insertions(+), 109 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/4694/4
-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908:     StartElection(NORMAL_ELECTION, reason);
> is there an interleaving in which we're already started a real election (bu
Yea, that looks possible, but I don't think it causes problems except for a potential extra election in a relatively rare case. If the pre-election for N+1 completes before we heard from N, then it'll bump its term to N+1 and call the real election for N+1. Then if N completes successfully, it'll just ignore the result coming from the prior term in DoElectionCallback.

Any suggestion for how we should handle this case differently? I think we already had a similar kind of case where it was possible to start an election for term N, not hear back, and then start one for N+1 under load, but given we do the backoff thing, eventually we should be spacing them out enough to succeed.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

LGTM

http://gerrit.cloudera.org:8080/#/c/4694/3/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1919:     StartElection(NORMAL_ELECTION, reason);
WARN_NOT_OK with prefix?


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/4694

to look at the new patch set (#3).

Change subject: KUDU-1365. Add leader pre-elections
......................................................................

KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

Cluster tests verify that this substantially reduces the election storms
after a node has a temporarily slow disk or otherwise freezes.

A new experimental flag 'raft_enable_pre_election' is introduced which
defaults the feature to on, but provides a safety valve to disable this
in case we find some bug after release.

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/tserver/tablet_service.cc
10 files changed, 261 insertions(+), 109 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/4694/3
-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908:     StartElection(NORMAL_ELECTION, reason);
is there an interleaving in which we're already started a real election (but haven't won it yet) when we get here? Something like:

1. Term N (follower): send a real election for term N
2. Election callback expires again
3. Term N (follower): send a pre-election for term N+1
4. Pre-election callback from term N+1 completes, even though we are still waiting on the result of the real election.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/4/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1385:     return RequestVoteRespondAlreadyVotedForOther(request, response);
I don't fully understand how B can reject the vote because it already voted. Isn't this the only place where it can happen?

In theory B shouldn't get to this block, as A sent a pre-election to current term + 1 (3) and B hand't bumped the term yet, meaning the check in LOG 1376 should fail.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908:     StartElection(NORMAL_ELECTION, reason);
> Yea, that looks possible, but I don't think it causes problems except for a
Just afraid that this interleaving will not be so unlikely with pre-elections. Since they require no IO, aren't they likely to go faster than regular elections? Maybe I'm missing a serialization point somewhere though.

Well we can always do nothing and see if this manifests itself in real scenarios. One thing we might consider if this becomes pathological (and I haven't fully thought this through), is to only do pre-elections when there isn't a real election going on. Since pre-elections are mostly for the sake of the partitioned/slow node that now can make sure that an election is called for when it comes back, pre-elections would still be helpful here, wheres if there just an election race where multiple replicas are trying to elect themselves pre-elections aren't helpful (maybe even worse?).


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/2/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1908:     StartElection(NORMAL_ELECTION, reason);
> Just afraid that this interleaving will not be so unlikely with pre-electio
There's not a guaranteed serialization point, but there is some kind of "softer" thing that helps make this less likely: the peers that are voting will be holding some lock while writing their "real election" vote to disk, and that would make the pre-election respond with "too busy". So, the pre-election that happens while a real election was in progress is quite likely to get denied due to hitting this code path.

+1 on trying this out with various stress tests and address it if we see this being a problem. Thanks for calling it out, though, I hadn't considered it.


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Mike Percy, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/4694

to look at the new patch set (#5).

Change subject: KUDU-1365. Add leader pre-elections
......................................................................

KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

Cluster tests verify that this substantially reduces the election storms
after a node has a temporarily slow disk or otherwise freezes.

A new experimental flag 'raft_enable_pre_election' is introduced which
defaults the feature to on, but provides a safety valve to disable this
in case we find some bug after release.

Tested this patch (along with the following series of cleanups) with:
 [1] 1000 loops of RaftConsensusITest.MultiThreadedInsertWithFailovers
 [2] 500 loops of raft-consensus-itest overall
 [3] 1000 loops of exactly_once_writes-itest

The above tests have usually been pretty good about finding bugs in
consensus in the past.

[1] http://dist-test.cloudera.org//job?job_id=todd.1476502616.16080
[2] http://dist-test.cloudera.org//job?job_id=todd.1476503111.17428
[3] http://dist-test.cloudera.org//job?job_id=todd.1476503268.18454

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/tserver/tablet_service.cc
10 files changed, 309 insertions(+), 144 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/4694/5
-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 4:

Sorry, made a typo in my scenario. Peer A has term 1, so when it sends pre-elections, it's for term 2 (the current term of peer B).

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 4:

Seeing one issue here.. with a 3-node config and one down, we can get stuck in this state:

peer A: current term 1, has ops through 1.41
peer B: current term 2, has ops through 1.25, voted for peer C in term 2

When peer A calls a pre-election for term 3, B rejects it because it already voted. When peer B calls a pre-election for term 3, peer A rejects it because it has a higher opid. So, they get "stuck".

This doesn't happen without pre-election because the voting process would ensure that peer A eventually ends up with a higher term than peer B.

I guess we need to do something such that a peer with an old term bumps its term up to at least the term of the voters that respond. Any thoughts?

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] WIP: KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: WIP: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 1:

submitted an itest x1000 loop here: http://dist-test.cloudera.org//job?job_id=todd.1476255551.11040

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 4:

| I guess we need to do something such that a peer with an old term bumps its term up to at least the term of the voters that respond. Any thoughts?

Yeah, that seems reasonable. If someone else has a higher term there's no point continuously trying to get elected with a lower one.

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 5: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4694/3/src/kudu/consensus/raft_consensus.cc
File src/kudu/consensus/raft_consensus.cc:

Line 1919:     StartElection(NORMAL_ELECTION, reason);
> WARN_NOT_OK with prefix?
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] KUDU-1365. Add leader pre-elections

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged.

Change subject: KUDU-1365. Add leader pre-elections
......................................................................


KUDU-1365. Add leader pre-elections

This implements the "pre-election" extension to the Raft algorithm.
The idea is that, before calling a leader election, a candidate first
sends a pre-election vote request to all voters. The voters respond
as they would have in a real vote, except they don't actually record
their vote.

Cluster tests verify that this substantially reduces the election storms
after a node has a temporarily slow disk or otherwise freezes.

A new experimental flag 'raft_enable_pre_election' is introduced which
defaults the feature to on, but provides a safety valve to disable this
in case we find some bug after release.

Tested this patch (along with the following series of cleanups) with:
 [1] 1000 loops of RaftConsensusITest.MultiThreadedInsertWithFailovers
 [2] 500 loops of raft-consensus-itest overall
 [3] 1000 loops of exactly_once_writes-itest

The above tests have usually been pretty good about finding bugs in
consensus in the past.

[1] http://dist-test.cloudera.org//job?job_id=todd.1476502616.16080
[2] http://dist-test.cloudera.org//job?job_id=todd.1476503111.17428
[3] http://dist-test.cloudera.org//job?job_id=todd.1476503268.18454

Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Reviewed-on: http://gerrit.cloudera.org:8080/4694
Tested-by: Kudu Jenkins
Reviewed-by: David Ribeiro Alves <dr...@apache.org>
---
M src/kudu/consensus/consensus.h
M src/kudu/consensus/consensus.proto
M src/kudu/consensus/leader_election-test.cc
M src/kudu/consensus/leader_election.cc
M src/kudu/consensus/leader_election.h
M src/kudu/consensus/raft_consensus.cc
M src/kudu/consensus/raft_consensus.h
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/tserver/tablet_service.cc
10 files changed, 309 insertions(+), 144 deletions(-)

Approvals:
  David Ribeiro Alves: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/4694
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifcfabd8c9ffe31f17ab768542a046426f656db43
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>