You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2017/10/06 22:10:41 UTC
kudu git commit: docs: clarify directory configs and usage
Repository: kudu
Updated Branches:
refs/heads/master 31d58522b -> 0e713aadb
docs: clarify directory configs and usage
Added a short blurb about how the various directory flags are used and
some notes about them (e.g. fs_wal_dir is ideally high-performance,
multiple fs_data_dirs stripes data, etc.).
Also updated the migration docs to clarify tool usage to avoid confusion
around `--fs_wal_dir=<master_data_dir>`.
Change-Id: Ia389da6421bddc3d19cbf7a49b4d3b8b59858f28
Reviewed-on: http://gerrit.cloudera.org:8080/8220
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/0e713aad
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/0e713aad
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/0e713aad
Branch: refs/heads/master
Commit: 0e713aadb9ba83df4d986dfe6bcdde1c1d30206b
Parents: 31d5852
Author: Andrew Wong <aw...@cloudera.com>
Authored: Thu Oct 5 11:08:21 2017 -0700
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Fri Oct 6 22:09:58 2017 +0000
----------------------------------------------------------------------
docs/administration.adoc | 45 +++++++++++++++++++++++--------------------
docs/configuration.adoc | 33 +++++++++++++++++++++----------
2 files changed, 47 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/0e713aad/docs/administration.adoc
----------------------------------------------------------------------
diff --git a/docs/administration.adoc b/docs/administration.adoc
index 26d2743..a35cb8f 100644
--- a/docs/administration.adoc
+++ b/docs/administration.adoc
@@ -211,12 +211,12 @@ how to migrate to a multi-master configuration.
WARNING: The workflow is unsafe for adding new masters to an existing multi-master configuration.
Do not use it for that purpose.
-WARNING: The workflow presupposes at least basic familiarity with Kudu configuration management. If
-using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
-
WARNING: All of the command line steps below should be executed as the Kudu UNIX user, typically
`kudu`.
+WARNING: The workflow presupposes at least basic familiarity with Kudu configuration management. If
+using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
+
==== Prepare for the migration
. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster
@@ -228,16 +228,17 @@ WARNING: All of the command line steps below should be executed as the Kudu UNIX
. Perform the following preparatory steps for the existing master:
* Identify and record the directory where the master's data lives. If using Kudu system packages,
the default value is /var/lib/kudu/master, but it may be customized via the `fs_wal_dir` and
- `fs_data_dirs` configuration parameter. Please note if you've set fs_data_dirs to some directories
- other than the value of fs_wal_dir, it should be explicitly included in every command below where
- fs_wal_dir is also included.
+ `fs_data_dirs` configuration parameters. Please note if you've set `fs_data_dirs` to some directories
+ other than the value of `fs_wal_dir`, it should be explicitly included in every command below where
+ `fs_wal_dir` is also included. For more information on configuring these directories, see the
+ link:configuration.html#directory_configuration[Kudu Configuration docs].
* Identify and record the port the master is using for RPCs. The default port value is 7051, but it
may have been customized using the `rpc_bind_addresses` configuration parameter.
* Identify the master's UUID. It can be fetched using the following command:
+
[source,bash]
----
-$ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
+$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
----
master_data_dir:: existing master's previously recorded data directory
+
@@ -277,8 +278,8 @@ bringing the cluster down for maintenance, and as such, it is highly recommended
+
[source,bash]
----
-$ kudu fs format --fs_wal_dir=<master_data_dir>
-$ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
+$ kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
+$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
----
+
master_data_dir:: new master's previously recorded data directory
@@ -302,7 +303,7 @@ f5624e05f40649b79a757629a69d061e
+
[source,bash]
----
-$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_data_dir> <tablet_id> <all_masters>
+$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <all_masters>
----
+
master_data_dir:: existing master's previously recorded data directory
@@ -332,7 +333,7 @@ port:: master's previously recorded RPC port number
+
[source,bash]
----
-$ kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>
+$ kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <existing_master>
----
+
master_data_dir:: new master's previously recorded data directory
@@ -419,14 +420,15 @@ WARNING: All of the command line steps below should be executed as the Kudu UNIX
. Perform the following preparatory steps for each live master:
* Identify and record the directory where the master's data lives. If using Kudu system packages,
the default value is /var/lib/kudu/master, but it may be customized via the `fs_wal_dir` and
- `fs_data_dirs` configuration parameter. Please note if you've set fs_data_dirs to some directories
- other than the value of fs_wal_dir, it should be explicitly included in every command below where
- fs_wal_dir is also included.
+ `fs_data_dirs` configuration parameters. Please note if you've set `fs_data_dirs` to some directories
+ other than the value of `fs_wal_dir`, it should be explicitly included in every command below where
+ `fs_wal_dir` is also included. For more information on configuring these directories, see the
+ link:configuration.html#directory_configuration[Kudu Configuration docs].
* Identify and record the master's UUID. It can be fetched using the following command:
+
[source,bash]
----
-$ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
+$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
----
master_data_dir:: live master's previously recorded data directory
+
@@ -441,14 +443,15 @@ $ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
. Perform the following preparatory steps for the reference master:
* Identify and record the directory where the master's data lives. If using Kudu system packages,
the default value is /var/lib/kudu/master, but it may be customized via the `fs_wal_dir` and
- `fs_data_dirs` configuration parameter. Please note if you've set fs_data_dirs to some directories
- other than the value of fs_wal_dir, it should be explicitly included in every command below where
- fs_wal_dir is also included.
+ `fs_data_dirs` configuration parameters. Please note if you've set `fs_data_dirs` to some directories
+ other than the value of `fs_wal_dir`, it should be explicitly included in every command below where
+ `fs_wal_dir` is also included. For more information on configuring these directories, see the
+ link:configuration.html#directory_configuration[Kudu Configuration docs].
* Identify and record the UUIDs of every master in the cluster, using the following command:
+
[source,bash]
----
-$ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_data_dir> <tablet_id> 2>/dev/null
+$ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> 2>/dev/null
----
master_data_dir:: reference master's previously recorded data directory
tablet_id:: must be the string `00000000000000000000000000000000`
@@ -471,7 +474,7 @@ $ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/var/lib/kudu/master
+
[source,bash]
----
-$ kudu fs format --fs_wal_dir=<master_data_dir> --uuid=<uuid>
+$ kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] --uuid=<uuid>
----
+
master_data_dir:: replacement master's previously recorded data directory
@@ -488,7 +491,7 @@ $ kudu fs format --fs_wal_dir=/var/lib/kudu/master --uuid=80a82c4b8a9f4c819bab74
+
[source,bash]
----
-$ kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <reference_master>
+$ kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <reference_master>
----
+
master_data_dir:: replacement master's previously recorded data directory
http://git-wip-us.apache.org/repos/asf/kudu/blob/0e713aad/docs/configuration.adoc
----------------------------------------------------------------------
diff --git a/docs/configuration.adoc b/docs/configuration.adoc
index 6fddf13..d397917 100644
--- a/docs/configuration.adoc
+++ b/docs/configuration.adoc
@@ -57,6 +57,25 @@ Take care when configuring undocumented flags, as not every possible
configuration has been tested, and undocumented options are not guaranteed to be
maintained in future releases.
+[[directory_configuration]]
+=== Directory Configurations
+Every Kudu node requires the specification of directory flags. The
+`--fs_wal_dir` configuration indicates where Kudu will place its write-ahead
+logs. It is recommended, although not necessary, that this directory is placed
+on a high-performance drive (one with high bandwidth and low latency, e.g. a
+solid-state drive). The `--fs_data_dirs` configuration indicates where Kudu
+will write its data blocks. This is a comma-separated list of directories; if
+multiple are specified, data will be striped across the directories. Kudu will
+also place metadata for each tablet in the first specified directory. If not
+specified, data blocks will be placed in the directory specified by
+`--fs_wal_dir`. Additionally, `--fs_wal_dir` may be the same as _one of_ the
+directories listed in `--fs_data_dirs`, but must not be a sub-directory of any
+of them.
+
+WARNING: Once these flags are set, they are difficult to change, often requiring
+the entire node to be rebuilt. For more details, see the
+link:administration.html#change_dir_config[Kudu Administration docs].
+
=== Configuring the Kudu Master
To see all available configuration options for the `kudu-master` executable, run it
with the `--help` option:
@@ -71,11 +90,8 @@ $ kudu-master --help
|--master_addresses | string | localhost | Comma-separated list of all the RPC
addresses for Master consensus-configuration. If not specified, assumes a standalone Master.
-|--fs_data_dirs | string | | Comma-separated list of
-directories where the Master will place its data blocks.
-|--fs_wal_dir | string | | The directory where the Master will
-place its write-ahead logs. May be the same as _one of_ the directories listed in
-`--fs_data_dirs`, but not a sub-directory of a data directory.
+|--fs_data_dirs | string | | List of directories where the Master will place its data blocks.
+|--fs_wal_dir | string | | The directory where the Master will place its write-ahead logs.
|--log_dir | string | /tmp | The directory to store Master log files.
|===
@@ -93,11 +109,8 @@ $ kudu-tserver --help
|===
| Flag | Valid Options | Default | Description
-|--fs_data_dirs | string | | Comma-separated list
-of directories where the Tablet Server will place its data blocks.
-|--fs_wal_dir | string | | The directory where the Tablet Server will
-place its write-ahead logs. May be the same as _one of_ the directories listed in
-`--fs_data_dirs`, but not a sub-directory of a data directory.
+|--fs_data_dirs | string | | List of directories where the Tablet Server will place its data blocks.
+|--fs_wal_dir | string | | The directory where the Tablet Server will place its write-ahead logs.
|--log_dir | string | /tmp | The directory to store Tablet Server log files
|--tserver_master_addrs | string | `127.0.0.1:7051` | Comma separated
addresses of the masters which the tablet server should connect to. The masters