You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ab...@apache.org on 2018/08/11 16:53:52 UTC
kudu git commit: [docs] Add changing master hostnames workflow
Repository: kudu
Updated Branches:
refs/heads/master 4fea7a3ca -> c866f4762
[docs] Add changing master hostnames workflow
Change-Id: I0df87d5e294d8b7bf5c7b8f94a63599ffd7ebe03
Reviewed-on: http://gerrit.cloudera.org:8080/11058
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/c866f476
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/c866f476
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/c866f476
Branch: refs/heads/master
Commit: c866f4762de0de2c78fb11fa8a08e8acc17443c3
Parents: 4fea7a3
Author: Attila Bukor <ab...@apache.org>
Authored: Sat Aug 11 11:24:30 2018 +0200
Committer: Attila Bukor <ab...@apache.org>
Committed: Sat Aug 11 16:53:22 2018 +0000
----------------------------------------------------------------------
docs/administration.adoc | 82 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 82 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/c866f476/docs/administration.adoc
----------------------------------------------------------------------
diff --git a/docs/administration.adoc b/docs/administration.adoc
index e5902f3..aebf382 100644
--- a/docs/administration.adoc
+++ b/docs/administration.adoc
@@ -644,6 +644,88 @@ To verify that all masters are working properly, perform the following sanity ch
* Run a Kudu system check (ksck) on the cluster using the `kudu` command line
tool. See <<ksck>> for more details.
+=== Changing the master hostnames
+
+To prevent long maintenance windows when replacing dead masters, DNS aliases should be used. If the
+cluster was set up without aliases, changing the host names can be done by following the below
+steps.
+
+==== Prepare for the hostname change
+
+. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster
+will be unavailable.
+
+. Note the UUID and RPC address of every master by visiting the `/masters` page of any master's web
+UI.
+
+. Stop all the Kudu processes in the entire cluster.
+
+. Set up the new hostnames to point to the masters and verify all servers and clients properly
+resolve them.
+
+==== Perform the hostname change
+
+. Rewrite each master’s Raft configuration with the following command, executed on all master hosts:
+----
+$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 00000000000000000000000000000000 <all_masters>
+----
+
+For example:
+
+----
+$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:new-master-name-1:7051 f5624e05f40649b79a757629a69d061e:new-master-name-2:7051 988d8ac6530f426cbe180be5ba52033d:new-master-name-3:7051
+----
+
+. Change the masters' gflagfile so the `master_addresses` parameter reflects the new hostnames.
+
+. Change the `tserver_master_addrs` parameter in the tablet servers' gflagfiles to the new
+hostnames.
+
+. Start up the masters.
+
+. To verify that all masters are working properly, perform the following sanity checks:
+
+.. Using a browser, visit each master's web UI. Look at the /masters page. All of the masters should
+ be listed there with one master in the LEADER role and the others in the FOLLOWER role. The
+ contents of /masters on each master should be the same.
+
+.. Run the below command to verify all masters are up and listening. The UUIDs
+should be the same and belong to the same master as before the hostname change:
++
+----
+$ sudo -u kudu kudu master list new-master-name-1:7051,new-master-name-2:7051,new-master-name-3:7051
+----
+
+. Start all of the tablet servers.
+
+. Run a Kudu system check (ksck) on the cluster using the `kudu` command line
+tool. See <<ksck>> for more details. After startup, some tablets may be
+unavailable as it takes some time to initialize all of them.
+
+. If you have Kudu tables that are accessed from Impala, update the HMS
+database manually in the underlying database that provides the storage for HMS.
+
+.. The following is an example SQL statement you should run in the HMS database:
++
+[source,sql]
+----
+UPDATE TABLE_PARAMS
+SET PARAM_VALUE =
+ 'new-master-name-1:7051,new-master-name-2:7051,new-master-name-3:7051'
+WHERE PARAM_KEY = 'kudu.master_addresses'
+AND PARAM_VALUE = 'master-1:7051,master-2:7051,master-3:7051';
+----
++
+.. In `impala-shell`, run:
++
+[source,bash]
+----
+INVALIDATE METADATA;
+----
++
+.. Verify updating the metadata worked by running a simple `SELECT` query on a
+Kudu-backed Impala table.
+
[[ksck]]
=== Checking Cluster Health with `ksck`