You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by da...@apache.org on 2016/10/07 18:18:15 UTC
[1/5] kudu git commit: docs: workflow for master migration
Repository: kudu
Updated Branches:
refs/heads/branch-1.0.x 8143d1ff4 -> 84ffbd14a
docs: workflow for master migration
Change-Id: I9b9c66505e0efd1f4aef80884346507d4fe08d9c
Reviewed-on: http://gerrit.cloudera.org:8080/4300
Reviewed-by: David Ribeiro Alves <dr...@apache.org>
Tested-by: Kudu Jenkins
(cherry picked from commit 1610b4ac42a9958dc2d79294cb860e1f998301ce)
Reviewed-on: http://gerrit.cloudera.org:8080/4657
Reviewed-by: Dan Burkert <da...@cloudera.com>
Tested-by: Dan Burkert <da...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/242de819
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/242de819
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/242de819
Branch: refs/heads/branch-1.0.x
Commit: 242de81992386c1d0a77720de0fb980c760d6476
Parents: 8143d1f
Author: Adar Dembo <ad...@cloudera.com>
Authored: Thu Sep 1 19:37:00 2016 -0700
Committer: Dan Burkert <da...@cloudera.com>
Committed: Fri Oct 7 18:04:29 2016 +0000
----------------------------------------------------------------------
docs/administration.adoc | 160 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 160 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/242de819/docs/administration.adoc
----------------------------------------------------------------------
diff --git a/docs/administration.adoc b/docs/administration.adoc
index 7e0d7e0..955bdbb 100644
--- a/docs/administration.adoc
+++ b/docs/administration.adoc
@@ -194,3 +194,163 @@ WARNING: Although metrics logging automatically rolls and compresses previous lo
not remove old ones. Since metrics logging can use significant amounts of disk space,
consider setting up a system utility to monitor space in the log directory and archive or
delete old segments.
+
+== Common Kudu workflows
+
+=== Migrating to Multiple Kudu Masters
+
+For high availability and to avoid a single point of failure, Kudu clusters should be created with
+multiple masters. Many Kudu clusters were created with just a single master, either for simplicity
+or because Kudu multi-master support was still experimental at the time. This workflow demonstrates
+how to migrate to a multi-master configuration.
+
+WARNING: The workflow is unsafe for adding new masters to an existing multi-master configuration.
+Do not use it for that purpose.
+
+WARNING: The workflow presupposes at least basic familiarity with Kudu configuration management. If
+using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
+
+WARNING: All of the command line steps below should be executed as the Kudu UNIX user, typically
+`kudu`.
+
+==== Prepare for the migration
+
+. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster
+ will be unavailable.
+
+. Decide how many masters to use. The number of masters should be odd. Three or five node master
+ configurations are recommendeded; they can tolerate one or two failures respectively.
+
+. Perform the following preparatory steps for the existing master:
+* Identify and record the directory where the master's data lives. If using Kudu system packages,
+ the default value is /var/lib/kudu/master, but it may be customized via the `fs_wal_dir`
+ configuration parameter.
+* Identify and record the port the master is using for RPCs. The default port value is 7051, but it
+ may have been customized using the `rpc_bind_addresses` configuration parameter.
+* Identify the master's UUID. It can be fetched using the following command:
++
+[source,bash]
+----
+$ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
+----
+master_data_dir:: existing master's previously recorded data directory
++
+[source,bash]
+.Example
+----
+$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+4aab798a69e94fab8d77069edff28ce0
+$
+----
++
+* Optional: configure a DNS alias for the master. The alias could be a DNS cname (if the machine
+ already has an A record in DNS), an A record (if the machine is only known by its IP address),
+ or an alias in /etc/hosts. Doing this simplifies recovering from permanent master failures
+ greatly, and is highly recommended. The alias should be an abstract representation of the
+ master (e.g. `master-1`).
+
+. Perform the following preparatory steps for each new master:
+* Choose an unused machine in the cluster. The master generates very little load so it can be
+ colocated with other data services or load-generating processes, though not with another Kudu
+ master from the same configuration.
+* Ensure Kudu is installed on the machine, either via system packages (in which case the `kudu` and
+ `kudu-master` packages should be installed), or via some other means.
+* Choose and record the directory where the master's data will live.
+* Choose and record the port the master should use for RPCs.
+* Optional: configure a DNS alias for the master (e.g. `master-2`, `master-3`, etc).
+
+==== Perform the migration
+
+. Stop all the Kudu processes in the entire cluster.
+
+. Format the data directory on each new master machine, and record the generated UUID. Use the
+ following command sequence:
++
+[source,bash]
+----
+$ kudu fs format --fs_wal_dir=<master_data_dir>
+$ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
+----
++
+master_data_dir:: new master's previously recorded data directory
++
+[source,bash]
+.Example
+----
+$ kudu fs format --fs_wal_dir=/var/lib/kudu/master
+$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+f5624e05f40649b79a757629a69d061e
+$
+----
+
+. If using CM, add the new Kudu master roles now, but do not start them. If using DNS aliases,
+ override the empty value of the `Master Address` parameter for each role (including the
+ existing master role) with that master's alias. Add the port number (separated by a colon) if
+ using a non-default RPC port value.
+
+. Rewrite the master's Raft configuration with the following command, executed on the existing
+ master machine:
++
+[source,bash]
+----
+$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_data_dir> <tablet_id> <all_masters>
+----
++
+master_data_dir:: existing master's previously recorded data directory
+tablet_id:: must be the string `00000000000000000000000000000000`
+all_masters:: space-separated list of masters, both new and existing. Each entry in the list must be
+ a string of the form `<uuid>:<hostname>:<port>`
+uuid::: master's previously recorded UUID
+hostname::: master's previously recorded hostname or alias
+port::: master's previously recorded RPC port number
++
+[source,bash]
+.Example
+----
+$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
+----
+
+. Start the existing master.
+
+. Copy the master data to each new master with the following command, executed on each new master
+ machine:
++
+[source,bash]
+----
+$ kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>
+----
++
+master_data_dir:: new master's previously recorded data directory
+tablet_id:: must be the string `00000000000000000000000000000000`
+existing_master:: RPC address of the existing master and must be a string of the form
+`<hostname>:<port>`
+hostname::: existing master's previously recorded hostname or alias
+port::: existing master's previously recorded RPC port number
++
+[source,bash]
+.Example
+----
+$ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 master-1:7051
+----
+
+. Start all of the new masters.
++
+WARNING: Skip the next step if using CM.
++
+. Modify the value of the `tserver_master_addrs` configuration parameter for each tablet server.
+ The new value must be a comma-separated list of masters where each entry is a string of the form
+ `<hostname>:<port>`
+hostname:: master's previously recorded hostname or alias
+port:: master's previously recorded RPC port number
+
+. Start all of the tablet servers.
+
+Congratulations, the cluster has now been migrated to multiple masters! To verify that all masters
+are working properly, consider performing the following sanity checks:
+
+* Using a browser, visit each master's web UI. Look at the /masters page. All of the masters should
+ be listed there with one master in the LEADER role and the others in the FOLLOWER role. The
+ contents of /masters on each master should be the same.
+
+* Run a Kudu system check (ksck) on the cluster using the `kudu` command line tool. Help for ksck
+ can be viewed via `kudu cluster ksck --help`.
[2/5] kudu git commit: [website] broken link to an adoc
Posted by da...@apache.org.
[website] broken link to an adoc
Change-Id: I8fe88073807eceb8927d6ca7ddf109bd0a9fd9d3
Reviewed-on: http://gerrit.cloudera.org:8080/4504
Reviewed-by: Will Berkeley <wd...@gmail.com>
Tested-by: Kudu Jenkins
(cherry picked from commit 36b71d2c5a6833da27db8c23be5180faaaad04ab)
Reviewed-on: http://gerrit.cloudera.org:8080/4658
Reviewed-by: Dan Burkert <da...@cloudera.com>
Tested-by: Dan Burkert <da...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/2869ac52
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/2869ac52
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/2869ac52
Branch: refs/heads/branch-1.0.x
Commit: 2869ac52200afd1427163f126232720338e60291
Parents: 242de81
Author: Jean-Daniel Cryans <jd...@apache.org>
Authored: Wed Sep 21 19:25:38 2016 -0700
Committer: Dan Burkert <da...@cloudera.com>
Committed: Fri Oct 7 18:04:52 2016 +0000
----------------------------------------------------------------------
docs/release_notes.adoc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/2869ac52/docs/release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc
index a97b5d6..90acb13 100644
--- a/docs/release_notes.adoc
+++ b/docs/release_notes.adoc
@@ -318,7 +318,7 @@ This affects RHEL/CentOS 6.8 and below. A fix is planned for RHEL/CentOS 6.9.
- link:http://kudu.apache.org[Kudu Website]
- link:http://github.com/apache/kudu[Kudu GitHub Repository]
- link:index.html[Kudu Documentation]
-- link:prior_release_notes.adoc[Release notes for older releases]
+- link:prior_release_notes.html[Release notes for older releases]
== Installation Options
[4/5] kudu git commit: Document Impala and Spark integration known
issues & limitations
Posted by da...@apache.org.
Document Impala and Spark integration known issues & limitations
Change-Id: I993a09a00f5ab0049fec95e967abc1740b44dc8d
Reviewed-on: http://gerrit.cloudera.org:8080/4443
Tested-by: Dan Burkert <da...@cloudera.com>
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
(cherry picked from commit 92f7c1914ab29061d324a9a38aa5bb05ca598d47)
Reviewed-on: http://gerrit.cloudera.org:8080/4660
Reviewed-by: Dan Burkert <da...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/6b30d7ea
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/6b30d7ea
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/6b30d7ea
Branch: refs/heads/branch-1.0.x
Commit: 6b30d7ea7da7dc54f0b593754b12c04febde0a0c
Parents: 427cf77
Author: Dan Burkert <da...@cloudera.com>
Authored: Fri Sep 16 14:16:36 2016 -0700
Committer: Dan Burkert <da...@cloudera.com>
Committed: Fri Oct 7 18:17:26 2016 +0000
----------------------------------------------------------------------
docs/developing.adoc | 14 ++++++++++++++
docs/kudu_impala_integration.adoc | 22 ++++++++++++++++++++++
2 files changed, 36 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/6b30d7ea/docs/developing.adoc
----------------------------------------------------------------------
diff --git a/docs/developing.adoc b/docs/developing.adoc
index b4d8604..8833369 100644
--- a/docs/developing.adoc
+++ b/docs/developing.adoc
@@ -151,6 +151,20 @@ kuduContext.tableExists("another_table")
kuduContext.deleteTable("unwanted_table")
----
+=== Spark Integration Known Issues and Limitations
+
+- The Kudu Spark integration is tested and developed against Spark 1.6 and Scala
+ 2.10.
+- Kudu tables with a name containing upper case or non-ascii characters must be
+ assigned an alternate name when registered as a temporary table.
+- Kudu tables with a column name containing upper case or non-ascii characters
+ may not be used with SparkSQL. Non-primary key columns may be renamed in Kudu
+ to work around this issue.
+- `NULL`, `NOT NULL`, `<>`, `OR`, `LIKE`, and `IN` predicates are not pushed to
+ Kudu, and instead will be evaluated by the Spark task.
+- Kudu does not support all types supported by Spark SQL, such as `Date`,
+ `Decimal` and complex types.
+
== Integration with MapReduce, YARN, and Other Frameworks
Kudu was designed to integrate with MapReduce, YARN, Spark, and other frameworks in
http://git-wip-us.apache.org/repos/asf/kudu/blob/6b30d7ea/docs/kudu_impala_integration.adoc
----------------------------------------------------------------------
diff --git a/docs/kudu_impala_integration.adoc b/docs/kudu_impala_integration.adoc
index e2fe89c..ec86c18 100755
--- a/docs/kudu_impala_integration.adoc
+++ b/docs/kudu_impala_integration.adoc
@@ -1083,3 +1083,25 @@ The examples above have only explored a fraction of what you can do with Impala
- View the link:http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/impala_langref.html[Impala SQL reference].
- Read about Impala internals or learn how to contribute to Impala on the link:https://github.com/cloudera/Impala/wiki[Impala Wiki].
- Read about the native link:installation.html#view_api[Kudu APIs].
+
+=== Known Issues and Limitations
+
+- Kudu tables with a name containing upper case or non-ascii characters must be
+ assigned an alternate name when used as an external table in Impala.
+- Kudu tables with a column name containing upper case or non-ascii characters
+ may not be used as an external table in Impala. Non-primary key columns may be
+ renamed in Kudu to work around this issue.
+- When creating a Kudu table, the `CREATE TABLE` statement must include the
+ primary key columns before other columns, in primary key order.
+- Kudu tables containing `UNIXTIME_MICROS`-typed columns may not be used as an
+ external table in Impala.
+- Impala can not create Kudu tables with `TIMESTAMP` or nested-typed columns.
+- Impala can not update values in primary key columns.
+- `NULL`, `NOT NULL`, `!=`, and `IN` predicates are not pushed to Kudu, and
+ instead will be evaluated by the Impala scan node.
+- Impala can not specify column encoding or compression during Kudu table
+ creation, or alter a columns encoding or compression.
+- Impala can not create Kudu tables with bounded range partitions, and can not
+ alter a table to add or remove range partitions.
+- When bulk writing to a Kudu table, performance may be improved by setting the
+ `batch_size` option (see <<kudu_impala_insert_bulk>>).
[5/5] kudu git commit: [doc] KUDU-1630 impala_tables.html link is
showing as text and not actual link
Posted by da...@apache.org.
[doc] KUDU-1630 impala_tables.html link is showing as text and not actual link
Missing space before the hyperlink was causing this issue.
Hyperlinked "Impala documentation" text to appropriate impala link.
Complete text being:
See the Impala documentation for more information about internal and external tables.
Please check the resultant html here:
https://github.com/ninadshr/sample_repo/blob/master/kudu_impala_integration.html
Change-Id: I25c1251b312586b38a558eb3365f60f7ff3a7ba0
Reviewed-on: http://gerrit.cloudera.org:8080/4483
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@cloudera.com>
Tested-by: Dan Burkert <da...@cloudera.com>
(cherry picked from commit 56cc69226fa6a053e6f038ac2b1d12b6fc533e37)
Reviewed-on: http://gerrit.cloudera.org:8080/4661
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/84ffbd14
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/84ffbd14
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/84ffbd14
Branch: refs/heads/branch-1.0.x
Commit: 84ffbd14a7087bd1123dde3fe87a77008e53c2a8
Parents: 6b30d7e
Author: ninadshr <ni...@cloudera.com>
Authored: Tue Sep 20 18:28:21 2016 -0400
Committer: Dan Burkert <da...@cloudera.com>
Committed: Fri Oct 7 18:17:53 2016 +0000
----------------------------------------------------------------------
docs/kudu_impala_integration.adoc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/84ffbd14/docs/kudu_impala_integration.adoc
----------------------------------------------------------------------
diff --git a/docs/kudu_impala_integration.adoc b/docs/kudu_impala_integration.adoc
index ec86c18..706b7b1 100755
--- a/docs/kudu_impala_integration.adoc
+++ b/docs/kudu_impala_integration.adoc
@@ -346,7 +346,8 @@ Impala, and dropping such a table does not drop the table from its source locati
(here, Kudu). Instead, it only removes the mapping between Impala and Kudu. This is
the mode used in the syntax provided by Kudu for mapping an existing table to Impala.
-See link:http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_tables.html
+See the
+http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_tables.html[Impala documentation]
for more information about internal and external tables.
=== Querying an Existing Kudu Table In Impala
[3/5] kudu git commit: [doc] KUDU-1627 Potentially Missing words in
the quick start doc
Posted by da...@apache.org.
[doc] KUDU-1627 Potentially Missing words in the quick start doc
Asciidoc was considering * as special markup for formatted
text and highlighting text in between.
Added formatting as `{asterisk}` to the markup to read the text as:
If the default projection generated by * does not
meet this requirement, the user should avoid using *
and explicitly
Change-Id: I1ab8f1a143b112ef4c43b1299ef67023cd74f98c
Reviewed-on: http://gerrit.cloudera.org:8080/4480
Reviewed-by: Dan Burkert <da...@cloudera.com>
Tested-by: Dan Burkert <da...@cloudera.com>
(cherry picked from commit 21aa44398c1ceb0f2c5b464ba3b36b7d85b7165b)
Reviewed-on: http://gerrit.cloudera.org:8080/4659
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/427cf770
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/427cf770
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/427cf770
Branch: refs/heads/branch-1.0.x
Commit: 427cf77087d6f1a2d96de73dad39cbdf2c7dfc6b
Parents: 2869ac5
Author: ninadshr <ni...@cloudera.com>
Authored: Tue Sep 20 13:47:48 2016 -0400
Committer: Dan Burkert <da...@cloudera.com>
Committed: Fri Oct 7 18:05:09 2016 +0000
----------------------------------------------------------------------
docs/quickstart.adoc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/427cf770/docs/quickstart.adoc
----------------------------------------------------------------------
diff --git a/docs/quickstart.adoc b/docs/quickstart.adoc
index cc5c328..976d098 100644
--- a/docs/quickstart.adoc
+++ b/docs/quickstart.adoc
@@ -165,8 +165,8 @@ Fetched 1 row(s) in 1.26s
====
For `CREATE TABLE ... AS SELECT` we currently require that the first columns that are
projected in the `SELECT` statement correspond to the Kudu table keys and are in the
-same order (`id` in the example above). If the default projection generated by `*`
-does not meet this requirement, the user should avoid using `*` and explicitly mention
+same order (`id` in the example above). If the default projection generated by `{asterisk}`
+does not meet this requirement, the user should avoid using `{asterisk}` and explicitly mention
the columns to project, in the correct order.
====
+