You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by gr...@apache.org on 2018/06/15 21:39:56 UTC
kudu git commit: Update website with doc improvements from master
Repository: kudu
Updated Branches:
refs/heads/gh-pages 4cdc8268c -> 8e3bce7ff
Update website with doc improvements from master
I generated updated docs and manually filtered out
any changes that are not relevant for the current or
past releases.
Change-Id: Ic846d497793bd16022cad7f0b185979bd68639a6
Reviewed-on: http://gerrit.cloudera.org:8080/10730
Reviewed-by: Will Berkeley <wd...@gmail.com>
Reviewed-by: Alex Rodoni <ar...@cloudera.com>
Reviewed-by: Mike Percy <mp...@apache.org>
Tested-by: Grant Henke <gr...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/8e3bce7f
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/8e3bce7f
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/8e3bce7f
Branch: refs/heads/gh-pages
Commit: 8e3bce7ffcc8dd0dea4fabfa6981b6bed5c72dd4
Parents: 4cdc826
Author: Grant Henke <gr...@apache.org>
Authored: Fri Jun 15 09:16:59 2018 -0500
Committer: Grant Henke <gr...@apache.org>
Committed: Fri Jun 15 21:37:34 2018 +0000
----------------------------------------------------------------------
docs/administration.html | 193 +++++++++++++--
docs/background_tasks.html | 6 +-
docs/command_line_tools.html | 6 +-
docs/command_line_tools_reference.html | 6 +-
docs/configuration.html | 12 +-
docs/configuration_reference.html | 6 +-
docs/configuration_reference_unsupported.html | 6 +-
docs/contributing.html | 40 ++-
docs/developing.html | 55 +++--
docs/export_control.html | 6 +-
docs/index.html | 6 +-
docs/installation.html | 35 ++-
docs/known_issues.html | 17 +-
docs/kudu-master_configuration_reference.html | 6 +-
...ter_configuration_reference_unsupported.html | 6 +-
docs/kudu-tserver_configuration_reference.html | 6 +-
...ver_configuration_reference_unsupported.html | 6 +-
docs/kudu_impala_integration.html | 41 +++-
docs/quickstart.html | 6 +-
docs/scaling_guide.html | 16 +-
docs/schema_design.html | 6 +-
docs/security.html | 6 +-
docs/transaction_semantics.html | 6 +-
docs/troubleshooting.html | 246 +++++++++++++++----
24 files changed, 612 insertions(+), 133 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/administration.html
----------------------------------------------------------------------
diff --git a/docs/administration.html b/docs/administration.html
index 9184d79..86b445d 100644
--- a/docs/administration.html
+++ b/docs/administration.html
@@ -2,7 +2,7 @@
title: Apache Kudu Administration
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -337,7 +337,7 @@ format exposed by the HTTP endpoint above.</p>
multiple masters. Many Kudu clusters were created with just a single master, either for simplicity
or because Kudu multi-master support was still experimental at the time. This workflow demonstrates
how to migrate to a multi-master configuration. It can also be used to migrate from two masters to
-three, with straightforward modifications.</p>
+three, with straightforward modifications. Note that the number of masters must be odd.</p>
</div>
<div class="admonitionblock warning">
<table>
@@ -359,6 +359,19 @@ three or more masters. Do not use it for that purpose.
<i class="fa icon-warning" title="Warning"></i>
</td>
<td class="content">
+An even number of masters doesn’t provide any benefit over having one fewer masters. This
+guide should always be used for migrating to three masters.
+</td>
+</tr>
+</table>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
All of the command line steps below should be executed as the Kudu UNIX user. The example
commands assume the Kudu Unix user is <code>kudu</code>, which is typical.
</td>
@@ -373,7 +386,8 @@ commands assume the Kudu Unix user is <code>kudu</code>, which is typical.
</td>
<td class="content">
The workflow presupposes at least basic familiarity with Kudu configuration management. If
-using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
+using vendor-specific tools (e.g. Cloudera Manager) the workflow also presupposes familiarity with
+it and the vendor’s instructions should be used instead as details may differ.
</td>
</tr>
</table>
@@ -803,7 +817,8 @@ more details on deploying with DNS aliases.
</td>
<td class="content">
The workflow presupposes at least basic familiarity with Kudu configuration management. If
-using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
+using vendor-specific tools (e.g. Cloudera Manager) the workflow also presupposes familiarity with
+it and the vendor’s instructions should be used instead as details may differ.
</td>
</tr>
</table>
@@ -1468,13 +1483,102 @@ directory configuration</a> to exclude the failed disk.</p>
</div>
</div>
<div class="sect2">
+<h3 id="disk_full_recovery"><a class="link" href="#disk_full_recovery">Recovering from Full Disks</a></h3>
+<div class="paragraph">
+<p>By default, Kudu reserves a small amount of space (1% by capacity) in its
+directories; Kudu considers a disk full if there is less free space available
+than the reservation. Kudu nodes can only tolerate running out of space on disks
+on which certain Kudu directories are mounted. For more information about the
+different Kudu directory types, see
+<a href="configuration.html#directory_configuration">Kudu Directory Configurations</a>.
+The table below describes this behavior for each type of directory. The behavior
+is uniform across masters and tablet servers.</p>
+</div>
+<table id="disk_full_behavior" class="tableblock frame-all grid-all spread">
+<caption class="title">Table 2. Kudu Full Disk Behavior</caption>
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Kudu Directory Type</th>
+<th class="tableblock halign-left valign-top">Crash on a Full Disk?</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Directory containing WALs</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Yes</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Directory containing tablet metadata</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Yes</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Directory containing data blocks only</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">No (see below)</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Prior to Kudu 1.7.0, Kudu stripes tablet data across all directories, and will
+avoid writing data to full directories. Kudu will crash if all data directories
+are full.</p>
+</div>
+<div class="paragraph">
+<p>In 1.7.0 and later, new tablets are assigned a disk group consisting of
+-fs_target_data_dirs_per_tablet data dirs (default 3). If Kudu is not configured
+with enough data directories for a full disk group, all data directories are
+used. When a data directory is full, Kudu will stop writing new data to it and
+each tablet that uses that data directory will write new data to other data
+directories within its group. If all data directories for a tablet are full, Kudu
+will crash. Periodically, Kudu will check if full data directories are still
+full, and will resume writing to those data directories if space has become
+available.</p>
+</div>
+<div class="paragraph">
+<p>If Kudu does crash because its data directories are full, freeing space on the
+full directories will allow the affected daemon to restart and resume writing.
+Note that it may be possible for Kudu to free some space by running</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-bash" data-lang="bash">$ sudo -u kudu kudu fs check --repair</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>but this command may also fail if there is too little space left.</p>
+</div>
+<div class="paragraph">
+<p>It’s also possible to allocate additional data directories to Kudu in order to
+increase the overall amount of storage available. See the documentation on
+<a href="#change_dir_config">updating a node’s directory configuration</a> for more
+information. Note that existing tablets will not use new data directories, so
+adding a new data directory does not resolve issues with full disks.</p>
+</div>
+</div>
+<div class="sect2">
<h3 id="tablet_majority_down_recovery"><a class="link" href="#tablet_majority_down_recovery">Bringing a tablet that has lost a majority of replicas back online</a></h3>
<div class="paragraph">
<p>If a tablet has permanently lost a majority of its replicas, it cannot recover
-automatically and operator intervention is required. The steps below may cause
-recent edits to the tablet to be lost, potentially resulting in permanent data
-loss. Only attempt the procedure below if it is impossible to bring
-a majority back online.</p>
+automatically and operator intervention is required. If the tablet servers
+hosting a majority of the replicas are down (i.e. ones reported as "TS
+unavailable" by ksck), they should be recovered instead if possible.</p>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+The steps below may cause recent edits to the tablet to be lost,
+potentially resulting in permanent data loss. Only attempt the procedure below
+if it is impossible to bring a majority back online.
+</td>
+</tr>
+</table>
</div>
<div class="paragraph">
<p>Suppose a tablet has lost a majority of its replicas. The first step in
@@ -1543,8 +1647,9 @@ re-replicate until the proper replication factor is restored. The unhealthy
replicas will be tombstoned by the master, causing their remaining data to be
deleted.</p>
</div>
-<div class="sect3">
-<h4 id="rebuilding_kudu"><a class="link" href="#rebuilding_kudu">Rebuilding a Kudu Filesystem Layout</a></h4>
+</div>
+<div class="sect2">
+<h3 id="rebuilding_kudu"><a class="link" href="#rebuilding_kudu">Rebuilding a Kudu Filesystem Layout</a></h3>
<div class="paragraph">
<p>In the event that critical files are lost, i.e. WALs or tablet-specific
metadata, all Kudu directories on the server must be deleted and rebuilt to
@@ -1623,6 +1728,63 @@ Kudu upon starting up.</p>
</ol>
</div>
</div>
+<div class="sect2">
+<h3 id="physical_backup"><a class="link" href="#physical_backup">Physical backups of an entire node</a></h3>
+<div class="paragraph">
+<p>As documented in the <a href="known_issues.html#_replication_and_backup_limitations">Known Issues and Limitations</a>,
+Kudu does not yet provide any built-in backup and restore functionality. However,
+it is possible to create a physical backup of a Kudu node (either tablet server
+or master) and restore it later.</p>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+The node to be backed up must be offline during the procedure, or else
+the backed up (or restored) data will be inconsistent.
+</td>
+</tr>
+</table>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+Certain aspects of the Kudu node (such as its hostname) are embedded in
+the on-disk data. As such, it’s not yet possible to restore a physical backup of
+a node onto another machine.
+</td>
+</tr>
+</table>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Stop all Kudu processes in the cluster. This prevents the tablets on the
+backed up node from being rereplicated elsewhere unnecessarily.</p>
+</li>
+<li>
+<p>If creating a backup, make a copy of the WAL, metadata, and data directories
+on each node to be backed up. It is important that this copy preserve all file
+attributes as well as sparseness.</p>
+</li>
+<li>
+<p>If restoring from a backup, delete the existing WAL, metadata, and data
+directories, then restore the backup via move or copy. As with creating a
+backup, it is important that the restore preserve all file attributes and
+sparseness.</p>
+</li>
+<li>
+<p>Start all Kudu processes in the cluster.</p>
+</li>
+</ol>
+</div>
</div>
</div>
</div>
@@ -1698,11 +1860,10 @@ Kudu upon starting up.</p>
<li><a href="#ksck">Checking Cluster Health with <code>ksck</code></a></li>
<li><a href="#change_dir_config">Changing Directory Configurations</a></li>
<li><a href="#disk_failure_recovery">Recovering from Disk Failure</a></li>
-<li><a href="#tablet_majority_down_recovery">Bringing a tablet that has lost a majority of replicas back online</a>
-<ul class="sectlevel3">
+<li><a href="#disk_full_recovery">Recovering from Full Disks</a></li>
+<li><a href="#tablet_majority_down_recovery">Bringing a tablet that has lost a majority of replicas back online</a></li>
<li><a href="#rebuilding_kudu">Rebuilding a Kudu Filesystem Layout</a></li>
-</ul>
-</li>
+<li><a href="#physical_backup">Physical backups of an entire node</a></li>
</ul>
</li>
</ul>
@@ -1721,6 +1882,10 @@ Kudu upon starting up.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/background_tasks.html
----------------------------------------------------------------------
diff --git a/docs/background_tasks.html b/docs/background_tasks.html
index cfa5df9..1d04701 100644
--- a/docs/background_tasks.html
+++ b/docs/background_tasks.html
@@ -2,7 +2,7 @@
title: Apache Kudu Background Maintenance Tasks
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -180,6 +180,10 @@ updates or upserts.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/command_line_tools.html
----------------------------------------------------------------------
diff --git a/docs/command_line_tools.html b/docs/command_line_tools.html
index 64f0dd1..82b4fe7 100644
--- a/docs/command_line_tools.html
+++ b/docs/command_line_tools.html
@@ -2,7 +2,7 @@
title: Command Hierarchy
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -2986,6 +2986,10 @@ Possible values: pretty, space, tsv, csv, and json</p></td>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/command_line_tools_reference.html
----------------------------------------------------------------------
diff --git a/docs/command_line_tools_reference.html b/docs/command_line_tools_reference.html
index 3f3ad2e..421a057 100644
--- a/docs/command_line_tools_reference.html
+++ b/docs/command_line_tools_reference.html
@@ -2,7 +2,7 @@
title: Apache Kudu Command Line Tools Reference
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -2977,6 +2977,10 @@ Possible values: pretty, space, tsv, csv, and json</p></td>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/configuration.html
----------------------------------------------------------------------
diff --git a/docs/configuration.html b/docs/configuration.html
index 6c52dd4..e9fde9f 100644
--- a/docs/configuration.html
+++ b/docs/configuration.html
@@ -2,7 +2,7 @@
title: Configuring Apache Kudu
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -87,7 +87,11 @@ logs. The <code>--fs_metadata_dir</code> configuration indicates where Kudu will
metadata for each tablet. It is recommended, although not necessary, that these
directories be placed on a high-performance drives with high bandwidth and low
latency, e.g. solid-state drives. If <code>--fs_metadata_dir</code> is not specified,
-metadata will be placed in the directory specified by <code>--fs_wal_dir</code>.</p>
+metadata will be placed in the directory specified by <code>--fs_wal_dir</code>. Since
+a Kudu node cannot tolerate the loss of its WAL or metadata directories, it
+may be wise to mirror the drives containing these directories in order to
+make recovering from a drive failure easier; however, mirroring may increase
+the latency of Kudu writes.</p>
</div>
<div class="paragraph">
<p>The <code>--fs_data_dirs</code> configuration indicates where Kudu will write its data
@@ -349,6 +353,10 @@ do not read this flag.</p></td>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/configuration_reference.html
----------------------------------------------------------------------
diff --git a/docs/configuration_reference.html b/docs/configuration_reference.html
index 1770b7b..9968614 100644
--- a/docs/configuration_reference.html
+++ b/docs/configuration_reference.html
@@ -2,7 +2,7 @@
title: Apache Kudu Configuration Reference
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -16270,6 +16270,10 @@ subject to being changed or removed without notice.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/configuration_reference_unsupported.html
----------------------------------------------------------------------
diff --git a/docs/configuration_reference_unsupported.html b/docs/configuration_reference_unsupported.html
index e0ef42a..d3e44f1 100644
--- a/docs/configuration_reference_unsupported.html
+++ b/docs/configuration_reference_unsupported.html
@@ -2,7 +2,7 @@
title: Apache Kudu Unsupported Configuration Flags
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -7314,6 +7314,10 @@ subject to being changed or removed without notice.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/contributing.html
----------------------------------------------------------------------
diff --git a/docs/contributing.html b/docs/contributing.html
index de3e798..1234e8d 100644
--- a/docs/contributing.html
+++ b/docs/contributing.html
@@ -2,7 +2,7 @@
title: Contributing to Apache Kudu
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -159,6 +159,40 @@ which you may find helpful.
</tr>
</table>
</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+You can add reviewers automatically for a patch by adding their GitHub
+username or associated email address to the remote branch name following with
+the "r" flag:
+</td>
+</tr>
+</table>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>git push gerrit HEAD:refs/for/master%r=githubuser,r=example@apache.org</pre>
+</div>
+</div>
+<div class="admonitionblock tip">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-tip" title="Tip"></i>
+</td>
+<td class="content">
+To find possible reviewer candidates for your commit, use git blame or git
+log to find out who are involved with the area you’re touching. It’s also a
+good idea to add as reviewer whoever is involved with the JIRA you’re working
+on.
+</td>
+</tr>
+</table>
+</div>
<div class="paragraph">
<p>Gerrit will add a change ID to your commit message and will create a Gerrit review,
whose URL will be emitted as part of the push reply. If desired, you can send a message
@@ -736,6 +770,10 @@ for guidelines about contributing to the official Kudu documentation.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/developing.html
----------------------------------------------------------------------
diff --git a/docs/developing.html b/docs/developing.html
index 1b7fb47..2f37460 100644
--- a/docs/developing.html
+++ b/docs/developing.html
@@ -2,7 +2,7 @@
title: Developing Applications With Apache Kudu
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -86,15 +86,18 @@ in <code>java/kudu-client/target/apidocs/index.html</code>.</p>
<div class="sectionbody">
<div class="paragraph">
<p>Several example applications are provided in the
-<a href="https://github.com/cloudera/kudu-examples">kudu-examples</a> Github
-repository. Each example includes a <code>README</code> that shows how to compile and run
-it. These examples illustrate correct usage of the Kudu APIs, as well as how to
-set up a virtual machine to run Kudu. The following list includes some of the
+<a href="https://github.com/apache/kudu/tree/master/examples">examples directory</a>
+of the Apache Kudu git repository. Each example includes a <code>README</code> that shows
+how to compile and run it. The following list includes some of the
examples that are available today. Check the repository itself in case this list goes
out of date.</p>
</div>
<div class="dlist">
<dl>
+<dt class="hdlist1"><code>cpp/example.cc</code></dt>
+<dd>
+<p>A simple C++ application which connects to a Kudu instance, creates a table, writes data to it, then drops the table.</p>
+</dd>
<dt class="hdlist1"><code>java/java-example</code></dt>
<dd>
<p>A simple Java application which connects to a Kudu instance, creates a table, writes data to it, then drops the table.</p>
@@ -115,12 +118,7 @@ generated by an external program, <code>dstat</code> in this case.</p>
</dd>
<dt class="hdlist1"><code>python/graphite-kudu</code></dt>
<dd>
-<p>An experimental plugin for using graphite-web with Kudu as a backend.</p>
-</dd>
-<dt class="hdlist1"><code>demo-vm-setup</code></dt>
-<dd>
-<p>Scripts to download and run a VirtualBox virtual machine with Kudu already installed.
-See <a href="quickstart.html">Quickstart</a> for more information.</p>
+<p>An example plugin for using graphite-web with Kudu as a backend.</p>
</dd>
</dl>
</div>
@@ -167,19 +165,22 @@ and using Impala with Kudu, including several <code>impala-shell</code> examples
Include the kudu-spark dependency using the --packages option:</p>
</div>
<div class="paragraph">
-<p>Use the kudu-spark_2.10 artifact if using Spark with Scala 2.10</p>
+<p>Use the kudu-spark_2.10 artifact if using Spark with Scala 2.10. Note that Spark 1 is no
+longer supported in Kudu starting from version 1.6.0. So in order to use Spark 1 integrated
+with Kudu, version 1.5.0 is the latest to go to.</p>
</div>
<div class="listingblock">
<div class="content">
-<pre class="highlight"><code>spark-shell --packages org.apache.kudu:kudu-spark_2.10:1.1.0</code></pre>
+<pre class="highlight"><code>spark-shell --packages org.apache.kudu:kudu-spark_2.10:1.5.0</code></pre>
</div>
</div>
<div class="paragraph">
-<p>Use kudu-spark2_2.11 artifact if using Spark 2 with Scala 2.11</p>
+<p>Use kudu-spark2_2.11 artifact if using Spark 2 with Scala 2.11. Spark 2 artifacts are available
+up to version 1.7.0.</p>
</div>
<div class="listingblock">
<div class="content">
-<pre class="highlight"><code>spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.1.0</code></pre>
+<pre class="highlight"><code>spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.7.0</code></pre>
</div>
</div>
<div class="paragraph">
@@ -192,17 +193,18 @@ import org.apache.kudu.client._
import collection.JavaConverters._
// Read a table from Kudu
-val df = sqlContext.read.options(Map("kudu.master" -> "kudu.master:7051","kudu.table" -> "kudu_table")).kudu
+val df = spark.read.options(Map("kudu.master" -> "kudu.master:7051",
+ "kudu.table" -> "kudu_table")).kudu
// Query using the Spark API...
-df.select("id").filter("id" >= 5).show()
+df.select("id").filter("id >= 5").show()
// ...or register a temporary table and use SQL
df.registerTempTable("kudu_table")
-val filteredDF = sqlContext.sql("select id from kudu_table where id >= 5").show()
+val filteredDF = spark.sql("select id from kudu_table where id >= 5").show()
// Use KuduContext to create, delete, or write to Kudu tables
-val kuduContext = new KuduContext("kudu.master:7051", sqlContext.sparkContext)
+val kuduContext = new KuduContext("kudu.master:7051", spark.sparkContext)
// Create a new Kudu table from a dataframe schema
// NB: No rows from the dataframe are inserted into the table
@@ -223,12 +225,15 @@ kuduContext.upsertRows(df, "test_table")
// Update data
val alteredDF = df.select("id", $"count" + 1)
-kuduContext.updateRows(filteredRows, "test_table"
+kuduContext.updateRows(filteredRows, "test_table")
-// Data can also be inserted into the Kudu table using the data source, though the methods on KuduContext are preferred
-// NB: The default is to upsert rows; to perform standard inserts instead, set operation = insert in the options map
+// Data can also be inserted into the Kudu table using the data source, though the methods on
+// KuduContext are preferred
+// NB: The default is to upsert rows; to perform standard inserts instead, set operation = insert
+// in the options map
// NB: Only mode Append is supported
-df.write.options(Map("kudu.master"-> "kudu.master:7051", "kudu.table"-> "test_table")).mode("append").kudu
+df.write.options(Map("kudu.master"-> "kudu.master:7051",
+ "kudu.table"-> "test_table")).mode("append").kudu
// Check for the existence of a Kudu table
kuduContext.tableExists("another_table")
@@ -430,6 +435,10 @@ using YARN and Spark in the future.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/export_control.html
----------------------------------------------------------------------
diff --git a/docs/export_control.html b/docs/export_control.html
index 38e3beb..d416f74 100644
--- a/docs/export_control.html
+++ b/docs/export_control.html
@@ -2,7 +2,7 @@
title: Export Control Notice
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -117,6 +117,10 @@ to provide secure authentication and TLS-protected transport.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/index.html
----------------------------------------------------------------------
diff --git a/docs/index.html b/docs/index.html
index 112ed2c..cae6fc5 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -2,7 +2,7 @@
title: Introducing Apache Kudu
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -433,6 +433,10 @@ formats using Impala, without the need to change your legacy systems.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/installation.html
----------------------------------------------------------------------
diff --git a/docs/installation.html b/docs/installation.html
index 43ec927..59d084a 100644
--- a/docs/installation.html
+++ b/docs/installation.html
@@ -2,7 +2,7 @@
title: Installing Apache Kudu
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -59,15 +59,29 @@ instead of following these instructions. For more information, see Cloudera̵
<div class="title">Hardware</div>
<ul>
<li>
-<p>One or more hosts to run Kudu masters. It is recommended to have either one master
-(no fault tolerance), three masters (can tolerate one failure), or five masters (two failures).</p>
+<p>One or more hosts to run Kudu masters. It is recommended to have either one master (no fault
+tolerance), or three masters (can tolerate one failure). The number of masters must be odd.</p>
</li>
<li>
-<p>One or more hosts to run Kudu tablet servers. When using replication, a minimum of
-three tablet servers is necessary.</p>
+<p>One or more hosts to run Kudu tablet servers. When using replication, a minimum of three tablet
+servers is necessary.</p>
</li>
</ul>
</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+A deployment with an even number of masters provides the same level of fault tolerance as a
+deployment with one fewer master. For example, both four-master and three-master deployments can
+only tolerate a single failure; two-master deployments cannot tolerate any failures.
+</td>
+</tr>
+</table>
+</div>
<div class="dlist">
<div class="title">Operating System Requirements</div>
<dl>
@@ -683,7 +697,7 @@ make -j4</code></pre>
<div class="content">
<pre>$ sudo zypper install autoconf automake curl cyrus-sasl-devel \
cyrus-sasl-gssapi flex gcc gcc-c++ gdb git java-1_8_0-openjdk-devel \
- krb5-devel libtool lsb-release make ntp openssl-devel patch \
+ krb5-devel krb5-server libtool lsb-release make ntp openssl-devel patch \
pkg-config python rsync unzip vim</pre>
</div>
</div>
@@ -789,7 +803,7 @@ $ sudo xcode-select --install</pre>
<div class="title">macOS Known Issues</div>
<div class="paragraph">
<p>Kudu support for macOS is experimental, and should only be used for development.
-See <a href="https://issues.cloudera.org/browse/KUDU-1219">macOS Limitations & Known Issues</a>
+See <a href="https://issues.apache.org/browse/KUDU-1219">macOS Limitations & Known Issues</a>
for more information.</p>
</div>
</td>
@@ -802,8 +816,7 @@ for more information.</p>
<p>Install the prerequisite libraries, if they are not installed.</p>
<div class="listingblock">
<div class="content">
-<pre>$ brew tap homebrew/dupes
-$ brew install autoconf automake cmake git krb5 libtool openssl pkg-config pstree</pre>
+<pre>$ brew install autoconf automake cmake git krb5 libtool openssl pkg-config pstree</pre>
</div>
</div>
</li>
@@ -1201,6 +1214,10 @@ $ sudo service kudu-tserver start</code></pre>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/known_issues.html
----------------------------------------------------------------------
diff --git a/docs/known_issues.html b/docs/known_issues.html
index 5481010..32d87a6 100644
--- a/docs/known_issues.html
+++ b/docs/known_issues.html
@@ -2,7 +2,7 @@
title: Known Issues and Limitations
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -189,17 +189,6 @@ and ideally more than 16GB when approaching the data and tablet <a href="#_scale
<p>Write ahead logs (WAL) can only be stored on one disk.</p>
</li>
<li>
-<p>Disk failures are not tolerated and tablets servers will crash as soon as one is detected.</p>
-</li>
-<li>
-<p>Failed disks with unrecoverable data require the formatting of all the Kudu data for
-that tablet server before it can be started again.</p>
-</li>
-<li>
-<p>Data directories cannot be added/removed; all must be reformatted to change the set
-of directories.</p>
-</li>
-<li>
<p>Tablet servers cannot be gracefully decommissioned.</p>
</li>
<li>
@@ -368,6 +357,10 @@ in <code>krb5.conf</code>.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/kudu-master_configuration_reference.html
----------------------------------------------------------------------
diff --git a/docs/kudu-master_configuration_reference.html b/docs/kudu-master_configuration_reference.html
index 99df8a4..98c917f 100644
--- a/docs/kudu-master_configuration_reference.html
+++ b/docs/kudu-master_configuration_reference.html
@@ -2,7 +2,7 @@
title: kudu-master Flags
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -4655,6 +4655,10 @@ being changed or removed without notice.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/kudu-master_configuration_reference_unsupported.html
----------------------------------------------------------------------
diff --git a/docs/kudu-master_configuration_reference_unsupported.html b/docs/kudu-master_configuration_reference_unsupported.html
index 3bfc0a1..3248085 100644
--- a/docs/kudu-master_configuration_reference_unsupported.html
+++ b/docs/kudu-master_configuration_reference_unsupported.html
@@ -2,7 +2,7 @@
title: kudu-master Unsupported Flags
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -4171,6 +4171,10 @@ subject to being changed or removed without notice.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/kudu-tserver_configuration_reference.html
----------------------------------------------------------------------
diff --git a/docs/kudu-tserver_configuration_reference.html b/docs/kudu-tserver_configuration_reference.html
index fc0e7a7..470cd1d 100644
--- a/docs/kudu-tserver_configuration_reference.html
+++ b/docs/kudu-tserver_configuration_reference.html
@@ -2,7 +2,7 @@
title: kudu-tserver Flags
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -4452,6 +4452,10 @@ being changed or removed without notice.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/kudu-tserver_configuration_reference_unsupported.html
----------------------------------------------------------------------
diff --git a/docs/kudu-tserver_configuration_reference_unsupported.html b/docs/kudu-tserver_configuration_reference_unsupported.html
index 6d79808..d619e6b 100644
--- a/docs/kudu-tserver_configuration_reference_unsupported.html
+++ b/docs/kudu-tserver_configuration_reference_unsupported.html
@@ -2,7 +2,7 @@
title: kudu-tserver Unsupported Flags
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:34:58 CDT'
+last_updated: 'Last updated 2018-06-14 12:04:30 PDT'
---
<!--
@@ -3751,6 +3751,10 @@ subject to being changed or removed without notice.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/kudu_impala_integration.html
----------------------------------------------------------------------
diff --git a/docs/kudu_impala_integration.html b/docs/kudu_impala_integration.html
index 91bd4b2..98c8ffa 100644
--- a/docs/kudu_impala_integration.html
+++ b/docs/kudu_impala_integration.html
@@ -2,7 +2,7 @@
title: Using Apache Kudu with Apache Impala
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -92,8 +92,7 @@ locations of the Kudu Master servers:</p>
<ul>
<li>
<p>Set the <code>--kudu_master_hosts=<master1>[:port],<master2>[:port],<master3>[:port]</code>
-flag in the Impala service configuration. If you are using Cloudera Manager,
-please refer to the appropriate Cloudera Manager documentation to do so.</p>
+flag in the Impala service configuration.</p>
</li>
</ul>
</div>
@@ -118,7 +117,7 @@ provide this configuration each time you create a table by specifying the
</td>
<td class="content">
This is only a small sub-set of Impala Shell functionality. For more details, see the
-<a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_impala_shell.html">Impala Shell</a> documentation.
+<a href="https://impala.apache.org/docs/build/html/topics/impala_impala_shell.html">Impala Shell</a> documentation.
</td>
</tr>
</table>
@@ -163,7 +162,7 @@ the mode used in the syntax provided by Kudu for mapping an existing table to Im
</div>
<div class="paragraph">
<p>See the
-<a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_tables.html">Impala documentation</a>
+<a href="https://impala.apache.org/docs/build/html/topics/impala_tables.html">Impala documentation</a>
for more information about internal and external tables.</p>
</div>
</div>
@@ -217,6 +216,24 @@ See <a href="#partitioning_tables">Partitioning Tables</a>. The table creation e
16 partitions by hashing the <code>id</code> column, for simplicity. See
<a href="#partitioning_rules_of_thumb">Partitioning Rules of Thumb</a> for guidelines on partitioning.</p>
</div>
+<div class="paragraph">
+<p>By default, Kudu tables created through Impala use a tablet replication factor of 3.
+To specify the replication factor for a Kudu table, add a <code>TBLPROPERTIES</code> clause
+to the <code>CREATE TABLE</code> statement as shown below where n is the replication factor
+you want to use:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-sql" data-lang="sql">TBLPROPERTIES ('kudu.num_tablet_replicas' = 'n')</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>A replication factor must be an odd number.</p>
+</div>
+<div class="paragraph">
+<p>Changing the <code>kudu.num_tablet_replicas</code> table property using ALTER TABLE currently
+has no effect.</p>
+</div>
<div class="sect3">
<h4 id="_code_create_table_as_select_code"><a class="link" href="#_code_create_table_as_select_code"><code>CREATE TABLE AS SELECT</code></a></h4>
<div class="paragraph">
@@ -733,7 +750,7 @@ in Impala.</p>
<ol class="arabic">
<li>
<p>If your data is not already in Impala, one strategy is to
-<a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_txtfile.html">import it from a text file</a>,
+<a href="https://impala.apache.org/docs/build/html/topics/impala_txtfile.html">import it from a text file</a>,
such as a TSV or CSV file.</p>
</li>
<li>
@@ -827,7 +844,7 @@ Kudu.
<div class="paragraph">
<p>You can also delete using more complex syntax. A comma in the <code>FROM</code> sub-clause is
one way that Impala specifies a join query. For more information about Impala joins,
-see <a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_joins.html" class="bare">http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_joins.html</a>.</p>
+see <a href="https://impala.apache.org/docs/build/html/topics/impala_joins.html" class="bare">https://impala.apache.org/docs/build/html/topics/impala_joins.html</a>.</p>
</div>
<div class="listingblock">
<div class="content">
@@ -963,13 +980,13 @@ data.</p>
<p>Learn about the <a href="http://impala.io">Impala project</a>.</p>
</li>
<li>
-<p>Read the <a href="http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/impala.html">Impala documentation</a>.</p>
+<p>Read the <a href="https://impala.apache.org/impala-docs.html">Impala documentation</a>.</p>
</li>
<li>
-<p>View the <a href="http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/impala_langref.html">Impala SQL reference</a>.</p>
+<p>View the <a href="https://impala.apache.org/docs/build/html/topics/impala_langref.html">Impala SQL reference</a>.</p>
</li>
<li>
-<p>Read about Impala internals or learn how to contribute to Impala on the <a href="https://github.com/cloudera/Impala/wiki">Impala Wiki</a>.</p>
+<p>Read about Impala internals or learn how to contribute to Impala on the <a href="https://cwiki.apache.org/confluence/display/IMPALA/Impala+Home">Impala Wiki</a>.</p>
</li>
<li>
<p>Read about the native <a href="installation.html#view_api">Kudu APIs</a>.</p>
@@ -1089,6 +1106,10 @@ for large tables.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/quickstart.html
----------------------------------------------------------------------
diff --git a/docs/quickstart.html b/docs/quickstart.html
index 6e6605f..42a2f01 100644
--- a/docs/quickstart.html
+++ b/docs/quickstart.html
@@ -2,7 +2,7 @@
title: Apache Kudu Quickstart
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -386,6 +386,10 @@ contain references to the previous VM’s SSH credentials. Remove any refere
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/scaling_guide.html
----------------------------------------------------------------------
diff --git a/docs/scaling_guide.html b/docs/scaling_guide.html
index b8d1a26..c160496 100644
--- a/docs/scaling_guide.html
+++ b/docs/scaling_guide.html
@@ -2,7 +2,7 @@
title: Apache Kudu Scaling Guide
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -417,6 +417,20 @@ generally peak when started and settle down thereafter.</p>
<a href="schema_design.html">Kudu Schema Design</a>
</li>
<li>
+<span class="active-toc">Kudu Scaling Guide</span>
+ <ul class="sectlevel1">
+<li><a href="#_terms">Terms</a></li>
+<li><a href="#_example_workload">Example Workload</a></li>
+<li><a href="#memory">Memory</a>
+<ul class="sectlevel2">
+<li><a href="#_verifying_if_a_memory_limit_is_sufficient">Verifying if a Memory Limit is sufficient</a></li>
+</ul>
+</li>
+<li><a href="#file_descriptors">File Descriptors</a></li>
+<li><a href="#threads">Threads</a></li>
+</ul>
+ </li>
+ <li>
<a href="security.html">Kudu Security</a>
</li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/schema_design.html
----------------------------------------------------------------------
diff --git a/docs/schema_design.html b/docs/schema_design.html
index 4855c37..fd426bf 100644
--- a/docs/schema_design.html
+++ b/docs/schema_design.html
@@ -2,7 +2,7 @@
title: Apache Kudu Schema Design
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -965,6 +965,10 @@ altered.</p>
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/security.html
----------------------------------------------------------------------
diff --git a/docs/security.html b/docs/security.html
index 92c9bf1..f327017 100644
--- a/docs/security.html
+++ b/docs/security.html
@@ -2,7 +2,7 @@
title: Apache Kudu Security
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -426,6 +426,10 @@ which require authentication or encryption.</p>
<a href="schema_design.html">Kudu Schema Design</a>
</li>
<li>
+
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
<span class="active-toc">Kudu Security</span>
<ul class="sectlevel1">
<li><a href="#_authentication">Authentication</a>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/transaction_semantics.html
----------------------------------------------------------------------
diff --git a/docs/transaction_semantics.html b/docs/transaction_semantics.html
index 44ba1fa..7f6ba3a 100644
--- a/docs/transaction_semantics.html
+++ b/docs/transaction_semantics.html
@@ -2,7 +2,7 @@
title: Transaction Semantics in Apache Kudu
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-17 09:37:29 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -518,6 +518,10 @@ frequently.
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>
http://git-wip-us.apache.org/repos/asf/kudu/blob/8e3bce7f/docs/troubleshooting.html
----------------------------------------------------------------------
diff --git a/docs/troubleshooting.html b/docs/troubleshooting.html
index d259ed6..bc005e3 100644
--- a/docs/troubleshooting.html
+++ b/docs/troubleshooting.html
@@ -2,7 +2,7 @@
title: Apache Kudu Troubleshooting
layout: default
active_nav: docs
-last_updated: 'Last updated 2018-03-21 12:01:40 CDT'
+last_updated: 'Last updated 2018-06-14 08:17:56 PDT'
---
<!--
@@ -44,11 +44,9 @@ Unpatched RHEL or CentOS 6.4 does not include a kernel with support for hole pun
<p>Ubuntu 14.04 includes version 3.13 of the Linux kernel, which supports hole punching.</p>
</li>
<li>
-<p>Newer versions of the EXT4 or XFS file systems support hole punching, but EXT3 does
-not. Older versions of XFS that do not support hole punching return a <code>EOPNOTSUPP</code>
-(operation not supported) error. Older versions of either EXT4 or XFS that do
-not support hole punching cause Kudu to emit an error message such as the following
-and to fail to start:</p>
+<p>Newer versions of the ext4 and xfs filesystems support hole punching. Older versions
+that do not support hole punching will cause Kudu to emit an error message such as the
+following and to fail to start:</p>
<div class="listingblock">
<div class="content">
<pre>Error during hole punch test. The log block manager requires a
@@ -62,6 +60,25 @@ details. Raw error message follows.</pre>
</li>
</ul>
</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+ext4 mountpoints may actually be backed by ext2 or ext3 formatted devices, which do not
+support hole punching. The hole punching test will fail when run on such filesystems. There
+are several different ways to determine whether an ext4 mountpoint is backed by an ext2,
+ext3, or ext4 formatted device; see <a href="https://unix.stackexchange.com/q/60723">this Stack
+Exchange post</a> for details.
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p><a href="administration.html#change_dir_config">Changing Directory Configurations</a> documentation.</p>
+</div>
<div class="paragraph">
<p>Without hole punching support, the log block manager is unsafe to use. It won’t
ever delete blocks, and will consume ever more space on disk.</p>
@@ -88,6 +105,25 @@ only be used for small-scale evaluation and development.
</div>
</div>
<div class="sect2">
+<h3 id="disk_issues"><a class="link" href="#disk_issues">Already present: FS layout already exists</a></h3>
+<div class="paragraph">
+<p>When Kudu starts, it checks each configured data directory, expecting either for all to be
+initialized or for all to be empty. If a server fails to start with a log message like</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre>Check failed: _s.ok() Bad status: Already present: FS layout already exists; not overwriting existing layout: FSManager roots already exist: /data0/kudu/data</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>then this precondition has failed. This could be because Kudu was configured with non-empty data
+directories on first startup, or because a previously-running, healthy Kudu process was restarted
+and at least one data directory was deleted or is somehow corrupted, perhaps because of a disk
+error. If in the latter situation, consult the
+<a href="administration.html#change_dir_config">Changing Directory Configurations</a> documentation.</p>
+</div>
+</div>
+<div class="sect2">
<h3 id="ntp"><a class="link" href="#ntp">NTP Clock Synchronization</a></h3>
<div class="paragraph">
<p>For the master and tablet server daemons, the server’s clock must be synchronized using NTP.
@@ -234,30 +270,42 @@ are currently acting as sources:</p>
</div>
<div class="listingblock">
<div class="content">
-<pre>$ ntpq -n -c opeers
+<pre>$ ntpq -nc lpeers
+ remote refid st t when poll reach delay offset jitter
+==============================================================================
+-108.59.2.24 130.133.1.10 2 u 13 64 1 71.743 0.373 0.016
++192.96.202.120 129.6.15.28 2 u 12 64 1 72.583 -0.426 0.028
+-69.10.161.7 204.26.59.157 3 u 11 64 1 15.741 2.641 0.021
+-173.255.206.154 45.56.123.24 3 u 10 64 1 43.502 0.199 0.029
+-69.195.159.158 128.138.140.44 2 u 9 64 1 53.885 -0.016 0.013
+*216.218.254.202 .CDMA. 1 u 6 64 1 1.475 -0.400 0.012
++129.250.35.250 249.224.99.213 2 u 7 64 1 1.342 -0.640 0.018
+ 45.76.244.193 216.239.35.4 2 u 6 64 1 17.380 -0.754 0.051
+ 69.89.207.199 212.215.1.157 2 u 5 64 1 57.796 -3.411 0.059
+ 171.66.97.126 .GPSs. 1 u 4 64 1 1.024 -0.374 0.018
+ 66.228.42.59 211.172.242.174 3 u 3 64 1 72.409 0.895 0.964
+ 91.189.89.198 17.253.34.125 2 u 2 64 1 135.195 -0.329 0.171
+ 162.210.111.4 216.218.254.202 2 u 1 64 1 28.570 0.693 0.306
+ 199.102.46.80 .GPS. 1 u 2 64 1 55.652 -0.039 0.019
+ 91.189.89.199 17.253.34.125 2 u 1 64 1 135.265 -0.413 0.037
+$ ntpq -nc opeers
remote local st t when poll reach delay offset disp
==============================================================================
- 0.0.0.0 0.0.0.0 16 p - 64 0 0.000 0.000 16000.0
- 0.0.0.0 0.0.0.0 16 p - 64 0 0.000 0.000 16000.0
- 0.0.0.0 0.0.0.0 16 p - 64 0 0.000 0.000 16000.0
- 0.0.0.0 0.0.0.0 16 p - 64 0 0.000 0.000 16000.0
- 0.0.0.0 0.0.0.0 16 p - 64 0 0.000 0.000 16000.0
--108.59.2.24 10.16.2.89 2 u 3 64 3 74.380 0.321 62.992
--208.82.104.205 10.16.2.89 2 u 5 64 3 52.654 -4.054 62.965
-#192.96.202.120 10.16.2.89 2 u 1 64 3 74.737 6.538 62.988
-#69.10.161.7 10.16.2.89 3 u 5 64 3 28.353 -1.967 62.960
--173.255.206.154 10.16.2.89 3 u - 64 3 42.906 -3.127 62.996
--69.195.159.158 10.16.2.89 2 u 1 64 3 52.543 -4.788 62.987
-*216.218.254.202 10.16.2.89 1 u 5 64 3 2.567 0.053 62.974
--129.250.35.250 10.16.2.89 2 u 3 64 3 2.603 0.256 62.985
-+45.76.244.193 10.16.2.89 2 u 5 64 3 19.522 0.188 62.969
--69.89.207.199 10.16.2.89 2 u 5 64 3 66.687 -0.395 62.967
--171.66.97.126 10.16.2.89 1 u 1 64 3 12.627 -3.572 62.963
-#66.228.42.59 10.16.2.89 4 u 1 64 3 72.143 4.034 62.971
- 91.189.89.198 10.16.2.89 2 u 5 64 3 135.329 3.069 3937.74
-#162.210.111.4 10.16.2.89 2 u - 64 3 29.572 6.849 62.966
-+199.102.46.80 10.16.2.89 1 u 3 64 3 57.022 0.111 63.386
- 91.189.89.199 10.16.2.89 2 u 4 64 3 138.269 3.228 3937.98</pre>
+-108.59.2.24 10.17.100.238 2 u 17 64 1 71.743 0.373 187.573
++192.96.202.120 10.17.100.238 2 u 16 64 1 72.583 -0.426 187.594
+-69.10.161.7 10.17.100.238 3 u 15 64 1 15.741 2.641 187.569
+-173.255.206.154 10.17.100.238 3 u 14 64 1 43.502 0.199 187.580
+-69.195.159.158 10.17.100.238 2 u 13 64 1 53.885 -0.016 187.561
+*216.218.254.202 10.17.100.238 1 u 10 64 1 1.475 -0.400 187.543
++129.250.35.250 10.17.100.238 2 u 11 64 1 1.342 -0.640 187.588
+ 45.76.244.193 10.17.100.238 2 u 10 64 1 17.380 -0.754 187.596
+ 69.89.207.199 10.17.100.238 2 u 9 64 1 57.796 -3.411 187.541
+ 171.66.97.126 10.17.100.238 1 u 8 64 1 1.024 -0.374 187.578
+ 66.228.42.59 10.17.100.238 3 u 7 64 1 72.409 0.895 187.589
+ 91.189.89.198 10.17.100.238 2 u 6 64 1 135.195 -0.329 187.584
+ 162.210.111.4 10.17.100.238 2 u 5 64 1 28.570 0.693 187.606
+ 199.102.46.80 10.17.100.238 1 u 4 64 1 55.652 -0.039 187.587
+ 91.189.89.199 10.17.100.238 2 u 3 64 1 135.265 -0.413 187.621</pre>
</div>
</div>
<div class="admonitionblock tip">
@@ -267,8 +315,8 @@ are currently acting as sources:</p>
<i class="fa icon-tip" title="Tip"></i>
</td>
<td class="content">
-Depending on the specific version of NTP, the correct command may be either
-<code>ntpq -n -c opeers</code> or <code>ntpq -n -c lpeers</code>.
+Both <code>opeers</code> and <code>lpeers</code> may be helpful as <code>lpeers</code> lists refid and
+jitter, while <code>lpeers</code> lists dispersion.
</td>
</tr>
</table>
@@ -320,12 +368,12 @@ results in a faster time synchronization when a machine restarts.</p>
<pre># Use my organization's internal NTP servers.
server ntp1.myorg.internal iburst
server ntp2.myorg.internal iburst
-# Provide several public pool servers from the US pool for
+# Provide several public pool servers for
# redundancy and robustness.
-server 0.pool.us.ntp.org iburst
-server 1.pool.us.ntp.org iburst
-server 2.pool.us.ntp.org iburst
-server 3.pool.us.ntp.org iburst</pre>
+server 0.pool.ntp.org iburst
+server 1.pool.ntp.org iburst
+server 2.pool.ntp.org iburst
+server 3.pool.ntp.org iburst</pre>
</div>
</div>
<div class="admonitionblock tip">
@@ -387,6 +435,61 @@ A common, though temporary, workaround for this is to restart NTP with one of th
</div>
</div>
<div class="sect1">
+<h2 id="disk_space_usage"><a class="link" href="#disk_space_usage">Disk Space Usage</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>When using the log block manager (the default on Linux), Kudu uses
+<a href="https://en.wikipedia.org/wiki/Sparse_file">sparse files</a> to store data. A
+sparse file has a different apparent size than the actual amount of disk space
+it uses. This means that some tools may inaccurately report the disk space
+used by Kudu. For example, the size listed by <code>ls -l</code> does not accurately
+reflect the disk space used by Kudu data files:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-bash" data-lang="bash">$ ls -lh /data/kudu/tserver/data
+total 117M
+-rw------- 1 kudu kudu 160M Mar 26 19:37 0b9807b8b17d48a6a7d5b16bf4ac4e6d.data
+-rw------- 1 kudu kudu 4.4K Mar 26 19:37 0b9807b8b17d48a6a7d5b16bf4ac4e6d.metadata
+-rw------- 1 kudu kudu 32M Mar 26 19:37 2f26eeacc7e04b65a009e2c9a2a8bd20.data
+-rw------- 1 kudu kudu 4.3K Mar 26 19:37 2f26eeacc7e04b65a009e2c9a2a8bd20.metadata
+-rw------- 1 kudu kudu 672M Mar 26 19:37 30a2dd2cd3554d8a9613f588a8d136ff.data
+-rw------- 1 kudu kudu 4.4K Mar 26 19:37 30a2dd2cd3554d8a9613f588a8d136ff.metadata
+-rw------- 1 kudu kudu 32M Mar 26 19:37 7434c83c5ec74ae6af5974e4909cbf82.data
+-rw------- 1 kudu kudu 4.3K Mar 26 19:37 7434c83c5ec74ae6af5974e4909cbf82.metadata
+-rw------- 1 kudu kudu 672M Mar 26 19:37 772d070347a04f9f8ad2ad3241440090.data
+-rw------- 1 kudu kudu 4.4K Mar 26 19:37 772d070347a04f9f8ad2ad3241440090.metadata
+-rw------- 1 kudu kudu 160M Mar 26 19:37 86e50a95531f46b6a79e671e6f5f4151.data
+-rw------- 1 kudu kudu 4.4K Mar 26 19:37 86e50a95531f46b6a79e671e6f5f4151.metadata
+-rw------- 1 kudu kudu 687 Mar 26 19:26 block_manager_instance</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Notice that the total size reported is 117MiB, while the first file’s size is
+listed as 160MiB. Adding the <code>-s</code> option to <code>ls</code> will cause <code>ls</code> to output the
+file’s disk space usage.</p>
+</div>
+<div class="paragraph">
+<p>The <code>du</code> and <code>df</code> utilities report the actual disk space usage by default.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-bash" data-lang="bash">$ du -h /data/kudu/tserver/data
+118M /data/kudu/tserver/data</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The apparent size can be shown with the <code>--apparent-size</code> flag to <code>du</code>.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-bash" data-lang="bash">$ du -h --apparent-size /data/kudu/tserver/data
+1.7G /data/kudu/tserver/data</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
<h2 id="crash_reporting"><a class="link" href="#crash_reporting">Reporting Kudu Crashes</a></h2>
<div class="sectionbody">
<div class="paragraph">
@@ -612,7 +715,7 @@ User stack:
</div>
<div class="paragraph">
<p>These traces can be useful for diagnosing root-cause latency issues when they are caused by systems
-below Kudu, such as disk controllers or file systems.</p>
+below Kudu, such as disk controllers or filesystems.</p>
</div>
</div>
<div class="sect2">
@@ -651,6 +754,17 @@ Generally, the recommended ratio of maintenance manager threads to data director
</li>
</ul>
</div>
+<div class="paragraph">
+<p>Finally, check the value of <code>--block_cache_capacity_mb</code>. This setting determines
+the maximum size of Kudu’s block cache. While a higher value can help with read
+and write performance, setting it too high (as a percentage of <code>--memory_limit_hard_bytes</code>)
+is harmful. Do not raise <code>--block_cache_capacity_mb</code> above <code>--memory_pressure_percentage</code>
+(default 60%) of <code>--memory_limit_hard_bytes</code>, as this will cause Kudu to flush
+aggressively even if write throughput is low. Keeping the block cache capacity
+below 50% of the memory pressure percentage times the hard limit is recommended.
+With the defaults, this means the <code>--block_cache_capacity_mb</code> should not exceed
+30% of <code>--memory_limit_hard_bytes</code>.</p>
+</div>
</div>
<div class="sect2">
<h3 id="heap_sampling"><a class="link" href="#heap_sampling">Heap Sampling</a></h3>
@@ -712,22 +826,18 @@ confidential or sensitive data.
</div>
</div>
<div class="sect2">
-<h3 id="disk_issues"><a class="link" href="#disk_issues">Disk Issues</a></h3>
+<h3 id="slow_dns_nscd"><a class="link" href="#slow_dns_nscd">Slow DNS Lookups and <code>nscd</code></a></h3>
<div class="paragraph">
-<p>When Kudu starts, it checks each configured data directory, expecting either for all to be
-initialized or for all to be empty. If a server fails to start with a log message like</p>
+<p>If the logs contain messages about slow name resolution like</p>
</div>
<div class="listingblock">
<div class="content">
-<pre>Check failed: _s.ok() Bad status: Already present: FS layout already exists; not overwriting existing layout: FSManager roots already exist: /data0/kudu/data</pre>
+<pre>W0926 11:19:01.339553 27231 net_util.cc:129] Time spent resolving address for kudu-tserver.example.com: real 4.647s user 0.000s sys 0.000s</pre>
</div>
</div>
<div class="paragraph">
-<p>then this precondition has failed. This could be because Kudu was configured with non-empty data
-directories on first startup, or because a previously-running, healthy Kudu process was restarted
-and at least one data directory was deleted or is somehow corrupted, perhaps because of a disk
-error. If in the latter situation, consult the
-<a href="administration.html#change_dir_config">Changing Directory Configurations</a> documentation.</p>
+<p>then it may help to run <code>nscd</code> (the name service cache daemon). Consult your
+operating system’s documentation for how to install and enable <code>nscd</code>.</p>
</div>
</div>
</div>
@@ -744,6 +854,42 @@ Hive in a format that’s unreadable to other tools, including Hive itself a
There is no workaround for Hive users. Spark users need to create temporary tables.</p>
</div>
</div>
+<div class="sect2">
+<h3 id="too_many_threads"><a class="link" href="#too_many_threads">Runtime error: Could not create thread: Resource temporarily unavailable (error 11)</a></h3>
+<div class="paragraph">
+<p>Users will encounter this error when Kudu is unable to create more threads,
+usually on versions of Kudu older than 1.7. It happens on tablet servers, and
+is a sign that the tablet server hosts too many tablet replicas. To fix the
+issue, users can raise the <code>nproc</code> ulimit as detailed in the documentation for
+their operating system or distribution. However, the better solution is to
+reduce the number of replicas on the tablet server. This may involve rethinking
+the table’s partitioning schema. For the recommended limits on number of
+replicas per tablet server, see the known issues and scaling limitations
+documentation for the appropriate Kudu release. The
+<a href="http://kudu.apache.org/releases/">releases page</a> has links to documentation
+for previous versions of Kudu; for the latest release, see the
+<a href="known_issues.html">known issues page</a>.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="tombstoned_or_stopped_tablets"><a class="link" href="#tombstoned_or_stopped_tablets">Tombstoned or STOPPED tablet replicas</a></h3>
+<div class="paragraph">
+<p>Users may notice some replicas on a tablet server are in a STOPPED state, and
+remain on the server indefinitely. These replicas are tombstones. A tombstone
+indicates that the tablet server once held a bona fide replica of its tablet.
+For example, if a tablet server goes down and its replicas are re-replicated
+elsewhere, if the tablet server rejoins the cluster its replicas will become
+tombstones. A tombstone will remain until the table it belongs to is deleted, or
+a new replica of the same tablet is placed on the tablet server. A count of
+tombstoned replicas and details of each one are available on the /tablets page
+of the tablet server web UI.</p>
+</div>
+<div class="paragraph">
+<p>The Raft consensus algorithm that Kudu uses for replication requires tombstones
+for correctness in certain rare situations. They consume minimal resources and
+hold no data. They must not be deleted.</p>
+</div>
+</div>
</div>
</div>
</div>
@@ -786,9 +932,11 @@ There is no workaround for Hive users. Spark users need to create temporary tabl
<li><a href="#_startup_errors">Startup Errors</a>
<ul class="sectlevel2">
<li><a href="#req_hole_punching">Errors During Hole Punching Test</a></li>
+<li><a href="#disk_issues">Already present: FS layout already exists</a></li>
<li><a href="#ntp">NTP Clock Synchronization</a></li>
</ul>
</li>
+<li><a href="#disk_space_usage">Disk Space Usage</a></li>
<li><a href="#crash_reporting">Reporting Kudu Crashes</a></li>
<li><a href="#_performance_troubleshooting">Performance Troubleshooting</a>
<ul class="sectlevel2">
@@ -797,12 +945,14 @@ There is no workaround for Hive users. Spark users need to create temporary tabl
<li><a href="#_kernel_stack_watchdog_traces">Kernel Stack Watchdog Traces</a></li>
<li><a href="#memory_limits">Memory Limits</a></li>
<li><a href="#heap_sampling">Heap Sampling</a></li>
-<li><a href="#disk_issues">Disk Issues</a></li>
+<li><a href="#slow_dns_nscd">Slow DNS Lookups and <code>nscd</code></a></li>
</ul>
</li>
<li><a href="#_issues_using_kudu">Issues using Kudu</a>
<ul class="sectlevel2">
<li><a href="#hive_handler">ClassNotFoundException: com.cloudera.kudu.hive.KuduStorageHandler</a></li>
+<li><a href="#too_many_threads">Runtime error: Could not create thread: Resource temporarily unavailable (error 11)</a></li>
+<li><a href="#tombstoned_or_stopped_tablets">Tombstoned or STOPPED tablet replicas</a></li>
</ul>
</li>
</ul>
@@ -817,6 +967,10 @@ There is no workaround for Hive users. Spark users need to create temporary tabl
</li>
<li>
+ <a href="scaling_guide.html">Kudu Scaling Guide</a>
+ </li>
+ <li>
+
<a href="security.html">Kudu Security</a>
</li>
<li>