[kudu-CR] [master] KUDU-2181 Procedure for copying sys catalog on adding master

["Bankim Bhavsar (Code Review)" <ge...@cloudera.org>]

Hello Alexey Serbin, Kudu Jenkins, Andrew Wong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16830

to look at the new patch set (#6).

Change subject: [master] KUDU-2181 Procedure for copying sys catalog on adding master
......................................................................

[master] KUDU-2181 Procedure for copying sys catalog on adding master

This change outlines procedure to copy system catalog for the newly
added master using existing CLI tools and the master ChangeConfig RPC.

Only functional change is to hookup the master runtime flag
--master_consensus_allow_status_msg_for_failed_peer to Raft consensus.
New master could go into a FAILED_RECOVERABLE state if the leader
master's system catalog WAL has been GC'ed. This change allows the
new master to be promoted after copying the system catalog externally.

Outline of the test procedure:
1) Runtime flag --master_consensus_allow_status_msg_for_failed_peer
must be turned on for existing masters.
2) Start the new master with
--master_address_add_new_master=<new-master-hostport> and
--master_addresses that contains itself and existing masters.
3) Invoke ChangeConfig to add the master.
4) Verify the new master is part of the Raft config even if it's a
LEARNER/NON_VOTER or goes into FAILED_RECOVERABLE state. If not,
above steps will have to be repeated.
5) If the new master is promoted to being a VOTER then following tablet
copy steps can be skipped.
6) Shutdown the new master.
7) Delete the system catalog on the new master.
8) Copy the system catalog from the leader master to the new master.
9) Bring up the new master.
10) Verify the new master is promoted as VOTER.
  If the new master doesn't get promoted to a VOTER then double check
  whether the new master is part of the Raft config for masters by
  running "kudu master list".
    - If yes, repeat procedure from step 6.
    - Else repeat the entire procedure

Change-Id: I142c1dec442ec72c38c5be9d62cdf270e441d6e3
---
M src/kudu/master/dynamic_multi_master-test.cc
M src/kudu/master/sys_catalog.cc
2 files changed, 294 insertions(+), 141 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/16830/6
-- 
To view, visit http://gerrit.cloudera.org:8080/16830
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I142c1dec442ec72c38c5be9d62cdf270e441d6e3
Gerrit-Change-Number: 16830
Gerrit-PatchSet: 6
Gerrit-Owner: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)