You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by jd...@apache.org on 2017/04/04 21:56:03 UTC
[1/4] kudu git commit: KUDU-1963. Avoid NPE and SSLException when a
TabletClient is closed during negotiation
Repository: kudu
Updated Branches:
refs/heads/branch-1.3.x 5020a5039 -> 0fada51ca
KUDU-1963. Avoid NPE and SSLException when a TabletClient is closed during negotiation
This fixes two related bugs which were triggered when a client called
close() on a TabletClient (e.g. because the end user of the API called
KuduClient#close()):
1) javax.net.ssl.SSLException: renegotiation attempted by peer; closing the connection
This was caused by the following interleaving in Netty:
T1 T2
------------------------ -----------------
TabletClient.close()
Channel.close()
SslHandler.close()
- starts a TLS "shutdown" message
- underlying SSLEngine is now in a
"closing" state call Channel.write()
call SSLEngine.write()
- gets NEEDS_UNWRAP
result
- interprets this as a
renegotiation and
throws an Exception
- call SSLEngine.unwrap() to get the
TLS message data to send
I wasn't able to write a unit test that triggered this reliably, since
TabletClient is too tightly coupled to ConnectionCache and
AsyncKuduClient. However, I was able to reproduce this reliably on an
Impala cluster by starting 100 threads sending short queries round-robin
to 9 impalads. Previously I saw the exception within a few seconds and
now I've run successfully for many minutes with no errors.
2) NullPointerException following above SSLException
When the above SSLException occurred, it called the
TabletClient#exceptionCaught() method, which called 'cleanup()' to fail
any outstanding RPCs. However, because the exception was being caught
because the client had already been explicitly closed, the
'rpcsInflight' and 'pendingRpcs' lists had already been set to null.
This caused an NPE as we tried to process those lists for a second time.
Another case which cased a similar NPE was that the NegotiationResult
could come after the close() had been called, in which case
'pendingRpcs' would already be null.
This patch fixes both issues as follows:
- if we've already explicitly closed the client, we ignore SSLExceptions
rather than logging them. Actually avoiding this race seemed to be
extremely complicated, so better to just ignore it.
- properly check whether the pendingRpcs/rpcsInFlight lists were already
null before processing them.
Change-Id: Ide91722446d01acb58e3e181e0d10932e023a043
Reviewed-on: http://gerrit.cloudera.org:8080/6541
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
(cherry picked from commit ac6e24fc2299430b5f70146324bef2d2e8a4b8bc)
Reviewed-on: http://gerrit.cloudera.org:8080/6543
Tested-by: Jean-Daniel Cryans <jd...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/ae25d1c1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/ae25d1c1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/ae25d1c1
Branch: refs/heads/branch-1.3.x
Commit: ae25d1c1fb8d00b8886711291235d0db1f161d33
Parents: 5020a50
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Apr 3 18:08:00 2017 -0700
Committer: Jean-Daniel Cryans <jd...@apache.org>
Committed: Tue Apr 4 21:42:27 2017 +0000
----------------------------------------------------------------------
.../org/apache/kudu/client/TabletClient.java | 23 ++++++++++++++++----
1 file changed, 19 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/ae25d1c1/java/kudu-client/src/main/java/org/apache/kudu/client/TabletClient.java
----------------------------------------------------------------------
diff --git a/java/kudu-client/src/main/java/org/apache/kudu/client/TabletClient.java b/java/kudu-client/src/main/java/org/apache/kudu/client/TabletClient.java
index e26e883..24e80f8 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/TabletClient.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/TabletClient.java
@@ -35,6 +35,7 @@ import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.locks.ReentrantLock;
import javax.annotation.concurrent.GuardedBy;
+import javax.net.ssl.SSLException;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
@@ -358,7 +359,12 @@ public class TabletClient extends SimpleChannelUpstreamHandler {
}
// Send the queued RPCs after dropping the lock, so we don't end up calling
// their callbacks/errbacks with the lock held.
- sendQueuedRpcs(queuedRpcs);
+ //
+ // queuedRpcs may be null in the case that we disconnected the client just
+ // at the same time as the Negotiator was finishing up.
+ if (queuedRpcs != null) {
+ sendQueuedRpcs(queuedRpcs);
+ }
return;
}
if (!(m instanceof CallResponse)) {
@@ -633,12 +639,14 @@ public class TabletClient extends SimpleChannelUpstreamHandler {
// for negotiation to complete.
if (pendingRpcs != null) {
rpcsToFail.addAll(pendingRpcs);
+ pendingRpcs = null;
}
- pendingRpcs = null;
// Similarly, we need to fail any that were already sent and in-flight.
- rpcsToFail.addAll(rpcsInflight.values());
- rpcsInflight = null;
+ if (rpcsInflight != null) {
+ rpcsToFail.addAll(rpcsInflight.values());
+ rpcsInflight = null;
+ }
} finally {
lock.unlock();
}
@@ -702,6 +710,13 @@ public class TabletClient extends SimpleChannelUpstreamHandler {
if (!closedByClient) {
LOG.info(getPeerUuidLoggingString() + "Lost connection to peer");
}
+ } else if (e instanceof SSLException && closedByClient) {
+ // There's a race in Netty where, when we call Channel.close(), it tries
+ // to send a TLS 'shutdown' message and enters a shutdown state. If another
+ // thread races to send actual data on the channel, then Netty will get a
+ // bit confused that we are trying to send data and misinterpret it as a
+ // renegotiation attempt, and throw an SSLException. So, we just ignore any
+ // SSLException if we've already attempted to close.
} else {
LOG.error(getPeerUuidLoggingString() + "Unexpected exception from downstream on " + c, e);
}
[2/4] kudu git commit: docs: Add a section for evolving gflags
Posted by jd...@apache.org.
docs: Add a section for evolving gflags
This patch adds a new section for evolving gflags in the documentation
and changes some of the XSLT logic to be more consistent.
It would be nice to replace the XSLT with a Python script.
Change-Id: I791d91e4f7ace4483958d26a60817857899fb474
Reviewed-on: http://gerrit.cloudera.org:8080/6489
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <wd...@gmail.com>
(cherry picked from commit 824b0f46dc3d1443ad10db1b07cb085cb5bd7a4e)
Reviewed-on: http://gerrit.cloudera.org:8080/6546
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/764b6c24
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/764b6c24
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/764b6c24
Branch: refs/heads/branch-1.3.x
Commit: 764b6c244f99ad5b4122b8b405802b5488da368e
Parents: ae25d1c
Author: Mike Percy <mp...@apache.org>
Authored: Fri Mar 24 19:27:21 2017 -0700
Committer: Jean-Daniel Cryans <jd...@apache.org>
Committed: Tue Apr 4 21:44:43 2017 +0000
----------------------------------------------------------------------
docs/configuration_reference.adoc | 6 +-
docs/configuration_reference_unsupported.adoc | 8 +--
docs/support/scripts/make_docs.sh | 4 +-
docs/support/xsl/gflags_to_asciidoc.xsl | 66 ++++++++++++++++++----
4 files changed, 64 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/764b6c24/docs/configuration_reference.adoc
----------------------------------------------------------------------
diff --git a/docs/configuration_reference.adoc b/docs/configuration_reference.adoc
index 630b0a5..eedc23a 100644
--- a/docs/configuration_reference.adoc
+++ b/docs/configuration_reference.adoc
@@ -28,9 +28,9 @@
:sectlinks:
:experimental:
-The contents of this file are generated from the output of the `--helpxml`
-flag for each binary, during the build of the documentation. Do not edit this file
-or the included files manually.
+// The contents of this file are generated from the output of the `--helpxml`
+// flag for each binary, during the build of the documentation. Do not edit
+// this file or the included files manually.
// This gets replaced by the script that builds the docs
@@CONFIGURATION_REFERENCE@@
http://git-wip-us.apache.org/repos/asf/kudu/blob/764b6c24/docs/configuration_reference_unsupported.adoc
----------------------------------------------------------------------
diff --git a/docs/configuration_reference_unsupported.adoc b/docs/configuration_reference_unsupported.adoc
index 20d0ecb..cbfd70f 100644
--- a/docs/configuration_reference_unsupported.adoc
+++ b/docs/configuration_reference_unsupported.adoc
@@ -27,12 +27,12 @@
:sectlinks:
:experimental:
-The contents of this file are generated from the output of the `--helpxml`
-flag for each binary, during the build of the documentation. Do not edit this file
-or the included files manually.
+// The contents of this file are generated from the output of the `--helpxml`
+// flag for each binary, during the build of the documentation. Do not edit
+// this file or the included files manually.
WARNING: These flags are unsupported and are included for informational purposes only.
-They are subject to change or be removed at any time.
+They are subject to being changed or removed at any time.
// This gets replaced by the script that builds the docs
@@CONFIGURATION_REFERENCE@@
http://git-wip-us.apache.org/repos/asf/kudu/blob/764b6c24/docs/support/scripts/make_docs.sh
----------------------------------------------------------------------
diff --git a/docs/support/scripts/make_docs.sh b/docs/support/scripts/make_docs.sh
index 8c94804..dcbc1f8 100755
--- a/docs/support/scripts/make_docs.sh
+++ b/docs/support/scripts/make_docs.sh
@@ -146,7 +146,7 @@ for binary in ${binaries[@]}; do
# Create the supported config reference
xsltproc \
--stringparam binary $binary \
- --stringparam support-level stable \
+ --stringparam support-level supported \
-o $GEN_DOC_DIR/${binary}_configuration_reference.adoc \
$SOURCE_ROOT/docs/support/xsl/gflags_to_asciidoc.xsl \
${GEN_DOC_DIR}/$binary.xml
@@ -184,7 +184,7 @@ cp $SOURCE_ROOT/docs/images/* "$OUTPUT_DIR/images/"
echo
echo ----------------------------
-echo "Docs built in $OUTPUT_DIR."
+echo "Docs built in $OUTPUT_DIR"
# If we're building the site, try to run Jekyll for them to make
# it a bit easier to quickly preview the results.
http://git-wip-us.apache.org/repos/asf/kudu/blob/764b6c24/docs/support/xsl/gflags_to_asciidoc.xsl
----------------------------------------------------------------------
diff --git a/docs/support/xsl/gflags_to_asciidoc.xsl b/docs/support/xsl/gflags_to_asciidoc.xsl
index ef79f88..a730251 100644
--- a/docs/support/xsl/gflags_to_asciidoc.xsl
+++ b/docs/support/xsl/gflags_to_asciidoc.xsl
@@ -59,16 +59,22 @@ limitations under the License.
:sectlinks:
:experimental:
-<xsl:if test="$support-level = 'stable'">
-[[<xsl:value-of select="$binary"/>_stable]]
+<!--start supported -->
+<xsl:if test="$support-level = 'supported'">
+[[<xsl:value-of select="$binary"/>_supported]]
= `<xsl:value-of select="$binary"/>` Flags
+[[<xsl:value-of select="$binary"/>_stable]]
== Stable Flags
Flags tagged `stable` and not `advanced` are safe to use for common
configuration tasks.
-<xsl:for-each select="flag"><xsl:if test="contains(tags, 'stable') and not(contains(tags, 'advanced')) and not(contains(tags, 'hidden'))">
+<xsl:for-each select="flag">
+ <xsl:if test="contains(tags, 'stable') and
+ not(contains(tags, 'advanced')) and
+ not(contains(tags, 'hidden')) and
+ not(contains(tags, 'unsafe'))">
[[<xsl:value-of select="$binary"/>_<xsl:value-of select="name"/>]]
=== `--<xsl:value-of select="name"/>`
@@ -82,7 +88,7 @@ configuration tasks.
|===
{nbsp}
-</xsl:if>
+ </xsl:if>
</xsl:for-each>
@@ -92,7 +98,11 @@ configuration tasks.
Flags tagged `stable` and `advanced` are supported, but should be considered
"expert" options and should be used carefully and after thorough testing.
-<xsl:for-each select="flag"><xsl:if test="contains(tags, 'stable') and contains(tags, 'advanced') and not(contains(tags, 'hidden'))">
+<xsl:for-each select="flag">
+ <xsl:if test="contains(tags, 'stable') and
+ contains(tags, 'advanced') and
+ not(contains(tags, 'hidden')) and
+ not(contains(tags, 'unsafe'))">
[[<xsl:value-of select="$binary"/>_<xsl:value-of select="name"/>]]
=== `--<xsl:value-of select="name"/>`
@@ -106,21 +116,55 @@ Flags tagged `stable` and `advanced` are supported, but should be considered
|===
{nbsp}
-</xsl:if>
+ </xsl:if>
</xsl:for-each>
+
+[[<xsl:value-of select="$binary"/>_evolving]]
+== Evolving Flags
+
+Flags tagged `evolving` (or not tagged with a stability tag) are not yet
+considered final, and while they may be useful for tuning, they are subject to
+being changed or removed without notice.
+
+<xsl:for-each select="flag">
+ <xsl:if test="not(contains(tags, 'stable')) and
+ not(contains(tags, 'experimental')) and
+ not(contains(tags, 'hidden')) and
+ not(contains(tags, 'unsafe'))">
+[[<xsl:value-of select="$binary"/>_<xsl:value-of select="name"/>]]
+=== `--<xsl:value-of select="name"/>`
+
+<xsl:value-of select="meaning"/>
+
+[cols="1h,3d", width="50%"]
+|===
+| Type | <xsl:value-of select="type"/>
+| Default | <xsl:choose><xsl:when test="default != ''">`<xsl:value-of select="default"/>`</xsl:when><xsl:otherwise>none</xsl:otherwise></xsl:choose>
+| Tags | <xsl:value-of select="tags"/>
+|===
+{nbsp}
+
+ </xsl:if>
+</xsl:for-each>
+
'''
</xsl:if>
-<!--end stable -->
+<!--end supported -->
<!-- start unsupported -->
<xsl:if test="$support-level = 'unsupported'">
[[<xsl:value-of select="$binary"/>_unsupported]]
= `<xsl:value-of select="$binary"/>` Unsupported Flags
-Flags marked `advanced` or `experimental` and not marked `stable`, or flags with no stability tag, are *unsupported* and are included
-for informational purposes only. They are subject to change or be removed without notice.
+Flags not marked `stable` or `evolving` are considered experimental and are
+*unsupported*. They are included here for informational purposes only and are
+subject to being changed or removed without notice.
-<xsl:for-each select="flag"><xsl:if test="not(contains(tags, 'stable')) and (contains(tags, 'advanced') or contains(tags, 'experimental')) and not(contains(tags, 'hidden'))">
+<xsl:for-each select="flag">
+ <xsl:if test="not(contains(tags, 'stable')) and
+ not(contains(tags, 'evolving')) and
+ not(contains(tags, 'hidden')) and
+ not(contains(tags, 'unsafe'))">
[[<xsl:value-of select="$binary"/>_<xsl:value-of select="name"/>]]
== `--<xsl:value-of select="name"/>`
@@ -132,7 +176,7 @@ for informational purposes only. They are subject to change or be removed withou
| Default | <xsl:choose><xsl:when test="default != ''">`<xsl:value-of select="default"/>`</xsl:when><xsl:otherwise>none</xsl:otherwise></xsl:choose>
| Tags | <xsl:value-of select="tags"/>
|===
-</xsl:if>
+ </xsl:if>
</xsl:for-each>
'''
</xsl:if>
[3/4] kudu git commit: docs: Move documentation style guide from user
guide to design-docs
Posted by jd...@apache.org.
docs: Move documentation style guide from user guide to design-docs
This isn't very important to the typical end-user and doesn't really
belong in the user guide. Let's move the way it's primarily viewed from
the web site to the git repo and link people to the GitHub rendering of
the doc from the contributing page of the user guide.
Change-Id: Ia9caa8767e2b3dd26cac11157b80a0b452d830c4
Reviewed-on: http://gerrit.cloudera.org:8080/6490
Tested-by: Mike Percy <mp...@apache.org>
Reviewed-by: Will Berkeley <wd...@gmail.com>
(cherry picked from commit 58e1d26d831a25835eb83aa21ac964f48792fdfb)
Reviewed-on: http://gerrit.cloudera.org:8080/6547
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Tested-by: Jean-Daniel Cryans <jd...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e237fc3a
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e237fc3a
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e237fc3a
Branch: refs/heads/branch-1.3.x
Commit: e237fc3a8a405f1371bb8f9d649389f154691c43
Parents: 764b6c2
Author: Mike Percy <mp...@apache.org>
Authored: Sat Mar 25 17:48:30 2017 -0700
Committer: Jean-Daniel Cryans <jd...@apache.org>
Committed: Tue Apr 4 21:51:26 2017 +0000
----------------------------------------------------------------------
docs/contributing.adoc | 6 +-
docs/design-docs/README.md | 1 +
docs/design-docs/doc-style-guide.adoc | 266 ++++++++++++++++++
docs/style_guide.adoc | 267 -------------------
docs/support/jekyll-templates/document.html.erb | 1 -
5 files changed, 271 insertions(+), 270 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/e237fc3a/docs/contributing.adoc
----------------------------------------------------------------------
diff --git a/docs/contributing.adoc b/docs/contributing.adoc
index f666b2c..0dd15fd 100644
--- a/docs/contributing.adoc
+++ b/docs/contributing.adoc
@@ -391,5 +391,7 @@ Commits which may affect performance should include before/after `perf-stat(1)`
== Documentation
-See link:style_guide.html[Documentation Style Guide] for guidelines about contributing
-to the official Kudu documentation.
+
+See the
+link:https://github.com/apache/kudu/blob/master/docs/design-docs/doc-style-guide.adoc[Documentation Style Guide]
+for guidelines about contributing to the official Kudu documentation.
http://git-wip-us.apache.org/repos/asf/kudu/blob/e237fc3a/docs/design-docs/README.md
----------------------------------------------------------------------
diff --git a/docs/design-docs/README.md b/docs/design-docs/README.md
index 6ceb3c3..e54704a 100644
--- a/docs/design-docs/README.md
+++ b/docs/design-docs/README.md
@@ -41,3 +41,4 @@ made.
| [Non-covering Range Partitions](non-covering-range-partitions.md) | Master, Client | [gerrit](http://gerrit.cloudera.org:8080/2772) |
| [Permanent failure handling of masters for Kudu 1.0](master-perm-failure-1.0.md) | Master | |
| [RPC Retry/Failover semantics](rpc-retry-and-failover.md) | Client/TS/Master | [gerrit](http://gerrit.cloudera.org:8080/2642) |
+| [Documentation Style Guide](doc-style-guide.adoc) | Documentation | |
http://git-wip-us.apache.org/repos/asf/kudu/blob/e237fc3a/docs/design-docs/doc-style-guide.adoc
----------------------------------------------------------------------
diff --git a/docs/design-docs/doc-style-guide.adoc b/docs/design-docs/doc-style-guide.adoc
new file mode 100644
index 0000000..e78b3dd
--- /dev/null
+++ b/docs/design-docs/doc-style-guide.adoc
@@ -0,0 +1,266 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+[[documentation_style_guide]]
+= Apache Kudu Documentation Style Guide
+
+:author: Kudu Team
+:icons: font
+:toc: left
+:toclevels: 3
+:doctype: book
+:backend: html5
+:sectlinks:
+:experimental:
+
+This document gives you the information you need to get started contributing to Kudu
+documentation. For code contribution guidelines, see
+link:https://kudu.apache.org/docs/contributing.html[Contributing to Kudu].
+
+== Asciidoc
+Kudu documentation is written in link:https://en.wikipedia.org/wiki/AsciiDoc[Asciidoc]
+and compiled into HTML and output using the link:http://asciidoctor.org/[Asciidoctor]
+toolchain. This provides several advantages. Among them:
+
+- Asciidoc is a superset of Markdown, so if you already know Markdown you can get
+started right away.
+- Github includes support for Asciidoc in its Atom editor, as well as real-time
+simplified HTML rendering.
+- Patch submissions are small and easy to review.
+
+== Code Standards
+Within reason, try to adhere to these standards:
+
+- 100 or fewer columns per line
+- 2 spaces rather than tabs for indentation
+- No more than 4 nested levels in the documentation if possible: `(Document -> Chapter
+-> Section -> Subsection)`
+- When possible, provide the language that a code listing is in, using the
+`[source,<language>]` macro. for example, `[source,sql]`
+- In general, do not indent Asciidoc, as indentation is significant. Code listings
+are one exception.
+
+== Building Documentation
+To build the documentation locally, you need the Asciidoctor Ruby application. To build the
+entire Kudu documentation set, change to the `docs/` directory and run:
+[source,bash]
+----
+asciidoctor -d book -D docs *.adoc
+----
+This builds the HTML output in a new _docs/_ directory within the current directory.
+Some content, such as the per-daemon configuration reference files, is not populated
+during a local build.
+
+To view the HTML, open _docs/index.html_ in your local browser.
+
+You can also build only a single chapter. such as _release_notes.adoc_, by passing its name instead.
+
+== Asciidoc Style Guide
+Asciidoc supports a lot of syntax that we do not need to use. When possible, stick
+with the following, adapted from the
+link:https://hbase.apache.org/book.html#_hbase_reference_guide_style_guide_and_cheat_sheet[HBase Reference Guide]:
+
+.AsciiDoc Cheat Sheet
+[cols="1,1,a",options="header"]
+|===
+| Element Type | Desired Rendering | How to do it
+| A paragraph | a paragraph | Just type some text with a blank line at the top and bottom.
+| Add line breaks within a paragraph without adding blank lines | Manual line breaks | This will break + at the plus sign. Or prefix the whole paragraph with a line containing '[%hardbreaks]'
+| Give a title to anything | Colored italic bold differently-sized text | .MyTitle (no space between the period and the words) on the line before the thing to be titled
+| In-Line Code or commands | monospace | \`text`
+| In-line literal content (things to be typed exactly as shown) | bold mono | \*\`typethis`*
+| In-line replaceable content (things to substitute with your own values) | bold italic mono | \*\_typesomething_*
+| Code blocks with highlighting | monospace, highlighted, preserve space |
+........
+[source,java]
+----
+ myAwesomeCode() {
+}
+----
+........
+| Code block included from a separate file | included just as though it were part of the main file |
+................
+[source,ruby]
+----
+\include::path/to/app.rb[]
+----
+................
+| Include only part of a separate file | Similar to Javadoc | See http://asciidoctor.org/docs/user-manual/#by-tagged-regions
+| File names, directory names, new terms | italic | \_hbase-default.xml_
+| External naked URLs | A link with the URL as link text |
+----
+http://www.google.com
+----
+
+| External URLs with text | A link with arbitrary link text |
+----
+link:http://www.google.com[Google]
+----
+
+| Create an internal anchor to cross-reference | not rendered |
+----
+[[anchor_name]]
+----
+| Cross-reference an existing anchor using its default title| an internal hyperlink using the element title if available, otherwise using the anchor name |
+----
+<<anchor_name>>
+----
+| Cross-reference an existing anchor using custom text | an internal hyperlink using arbitrary text |
+----
+<<anchor_name,Anchor Text>>
+----
+| A block image | The image with alt text |
+----
+image::sunset.jpg[Alt Text]
+----
+(put the image in the src/main/site/resources/images directory)
+| An inline image | The image with alt text, as part of the text flow |
+----
+image:sunset.jpg [Alt Text]
+----
+(only one colon)
+| Link to a remote image | show an image hosted elsewhere |
+----
+image::http://inkscape.org/doc/examples/tux.svg[Tux,250,350]
+----
+(or `image:`)
+| Add dimensions or a URL to the image | depends | inside the brackets after the alt text, specify width, height and/or link="http://my_link.com"
+| A footnote | subscript link which takes you to the footnote |
+----
+Some text.footnote:[The footnote text.]
+----
+| A note or warning with no title | The admonition image followed by the admonition |
+----
+NOTE: My note here
+----
+
+----
+WARNING: My warning here
+----
+| A complex note | The note has a title and/or multiple paragraphs and/or code blocks or lists, etc |
+........
+.The Title
+[NOTE]
+====
+Here is the note text.
+Everything until the second set of four equals signs
+is part of the note.
+----
+some source code
+----
+====
+........
+| Bullet lists | bullet lists |
+----
+* list item 1
+----
+(see http://asciidoctor.org/docs/user-manual/#unordered-lists)
+| Numbered lists | numbered list |
+----
+. list item 2
+----
+(see http://asciidoctor.org/docs/user-manual/#ordered-lists)
+| Checklists | Checked or unchecked boxes |
+Checked:
+----
+- [*]
+----
+Unchecked:
+----
+- [ ]
+----
+| Multiple levels of lists | bulleted or numbered or combo |
+----
+. Numbered (1), at top level
+* Bullet (2), nested under 1
+* Bullet (3), nested under 1
+. Numbered (4), at top level
+* Bullet (5), nested under 4
+** Bullet (6), nested under 5
+- [x] Checked (7), at top level
+----
+| Labelled lists / variablelists | a list item title or summary followed by content |
+----
+Title:: content
+
+Title::
+ content
+----
+|GUI menu cascades | bold text with arrows to show levels | Use an ASCII arrow.
+........
+*File -> Print*
+........
+renders like *File -> Print*
+| Sidebars, quotes, or other blocks of text | a block of text, formatted differently from the default | Delimited using different delimiters, see http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary. Some of the examples above use delimiters like \...., ----,====.
+........
+[example]
+====
+This is an example block.
+====
+
+[source]
+----
+This is a source block.
+----
+
+[note]
+====
+This is a note block.
+====
+
+[quote]
+____
+This is a quote block.
+____
+........
+
+*If you want to insert literal Asciidoc content that keeps being interpreted, when in doubt, use eight dots as the delimiter at the top and bottom.*
+| Nested Sections | chapter, section, sub-section, etc |
+----
+= Book (or chapter if the chapter can be built alone, see leveloffset below)
+
+== Chapter (or section if the chapter is standalone)
+
+=== Section (or subsection, etc)
+
+==== Subsection
+----
+
+and so on up to 6 levels (think carefully about going deeper than 4 levels, maybe you can just titled paragraphs or lists instead). Note that you can include a book inside another book by adding the `:leveloffset:+1` macro directive directly before your include, and resetting it to 0 directly after. See the _book.adoc_ source for examples, as this is how this guide handles chapters. *Don't do it for prefaces, glossaries, appendixes, or other special types of chapters.*
+
+| Include one file from another | Content is included as though it were inline |
+
+----
+include::[/path/to/file.adoc]
+----
+
+For plenty of examples. see _docs/docs.adoc_.
+| A table | a table | See http://asciidoctor.org/docs/user-manual/#tables. Generally rows are separated by newlines and columns by pipes
+| Comment out a single line | A line is skipped during rendering |
+`+//+ This line won't show up`
+| Comment out a block | A section of the file is skipped during rendering |
+----
+////
+Nothing between the slashes will show up.
+////
+----
+| Highlight text for review | text shows up with yellow background |
+----
+Test between #hash marks# is highlighted yellow.
+----
+|===
+
http://git-wip-us.apache.org/repos/asf/kudu/blob/e237fc3a/docs/style_guide.adoc
----------------------------------------------------------------------
diff --git a/docs/style_guide.adoc b/docs/style_guide.adoc
deleted file mode 100644
index 3c25423..0000000
--- a/docs/style_guide.adoc
+++ /dev/null
@@ -1,267 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-[[documentation_style_guide]]
-= Apache Kudu Documentation Style Guide
-
-:author: Kudu Team
-:imagesdir: ./images
-:icons: font
-:toc: left
-:toclevels: 3
-:doctype: book
-:backend: html5
-:sectlinks:
-:experimental:
-
-This document gives you the information you need to get started contributing to Kudu
-documentation. For code contribution guidelines, see
-link:contributing.html[Contributing to Kudu].
-
-== Asciidoc
-Kudu documentation is written in link:https://en.wikipedia.org/wiki/AsciiDoc[Asciidoc]
-and compiled into HTML and output using the link:http://asciidoctor.org/[Asciidoctor]
-toolchain. This provides several advantages. Among them:
-
-- Asciidoc is a superset of Markdown, so if you already know Markdown you can get
-started right away.
-- Github includes support for Asciidoc in its Atom editor, as well as real-time
-simplified HTML rendering.
-- Patch submissions are small and easy to review.
-
-== Code Standards
-Within reason, try to adhere to these standards:
-
-- 100 or fewer columns per line
-- 2 spaces rather than tabs for indentation
-- No more than 4 nested levels in the documentation if possible: `(Document -> Chapter
--> Section -> Subsection)`
-- When possible, provide the language that a code listing is in, using the
-`[source,<language>]` macro. for example, `[source,sql]`
-- In general, do not indent Asciidoc, as indentation is significant. Code listings
-are one exception.
-
-== Building Documentation
-To build the documentation locally, you need the Asciidoctor Ruby application. To build the
-entire Kudu documentation set, change to the `docs/` directory and run:
-[source,bash]
-----
-asciidoctor -d book -D docs *.adoc
-----
-This builds the HTML output in a new _docs/_ directory within the current directory.
-Some content, such as the per-daemon configuration reference files, is not populated
-during a local build.
-
-To view the HTML, open _docs/index.html_ in your local browser.
-
-You can also build only a single chapter. such as _release_notes.adoc_, by passing its name instead.
-
-== Asciidoc Style Guide
-Asciidoc supports a lot of syntax that we do not need to use. When possible, stick
-with the following, adapted from the
-link:https://hbase.apache.org/book.html#_hbase_reference_guide_style_guide_and_cheat_sheet[HBase Reference Guide]:
-
-.AsciiDoc Cheat Sheet
-[cols="1,1,a",options="header"]
-|===
-| Element Type | Desired Rendering | How to do it
-| A paragraph | a paragraph | Just type some text with a blank line at the top and bottom.
-| Add line breaks within a paragraph without adding blank lines | Manual line breaks | This will break + at the plus sign. Or prefix the whole paragraph with a line containing '[%hardbreaks]'
-| Give a title to anything | Colored italic bold differently-sized text | .MyTitle (no space between the period and the words) on the line before the thing to be titled
-| In-Line Code or commands | monospace | \`text`
-| In-line literal content (things to be typed exactly as shown) | bold mono | \*\`typethis`*
-| In-line replaceable content (things to substitute with your own values) | bold italic mono | \*\_typesomething_*
-| Code blocks with highlighting | monospace, highlighted, preserve space |
-........
-[source,java]
-----
- myAwesomeCode() {
-}
-----
-........
-| Code block included from a separate file | included just as though it were part of the main file |
-................
-[source,ruby]
-----
-\include::path/to/app.rb[]
-----
-................
-| Include only part of a separate file | Similar to Javadoc | See http://asciidoctor.org/docs/user-manual/#by-tagged-regions
-| File names, directory names, new terms | italic | \_hbase-default.xml_
-| External naked URLs | A link with the URL as link text |
-----
-http://www.google.com
-----
-
-| External URLs with text | A link with arbitrary link text |
-----
-link:http://www.google.com[Google]
-----
-
-| Create an internal anchor to cross-reference | not rendered |
-----
-[[anchor_name]]
-----
-| Cross-reference an existing anchor using its default title| an internal hyperlink using the element title if available, otherwise using the anchor name |
-----
-<<anchor_name>>
-----
-| Cross-reference an existing anchor using custom text | an internal hyperlink using arbitrary text |
-----
-<<anchor_name,Anchor Text>>
-----
-| A block image | The image with alt text |
-----
-image::sunset.jpg[Alt Text]
-----
-(put the image in the src/main/site/resources/images directory)
-| An inline image | The image with alt text, as part of the text flow |
-----
-image:sunset.jpg [Alt Text]
-----
-(only one colon)
-| Link to a remote image | show an image hosted elsewhere |
-----
-image::http://inkscape.org/doc/examples/tux.svg[Tux,250,350]
-----
-(or `image:`)
-| Add dimensions or a URL to the image | depends | inside the brackets after the alt text, specify width, height and/or link="http://my_link.com"
-| A footnote | subscript link which takes you to the footnote |
-----
-Some text.footnote:[The footnote text.]
-----
-| A note or warning with no title | The admonition image followed by the admonition |
-----
-NOTE: My note here
-----
-
-----
-WARNING: My warning here
-----
-| A complex note | The note has a title and/or multiple paragraphs and/or code blocks or lists, etc |
-........
-.The Title
-[NOTE]
-====
-Here is the note text.
-Everything until the second set of four equals signs
-is part of the note.
-----
-some source code
-----
-====
-........
-| Bullet lists | bullet lists |
-----
-* list item 1
-----
-(see http://asciidoctor.org/docs/user-manual/#unordered-lists)
-| Numbered lists | numbered list |
-----
-. list item 2
-----
-(see http://asciidoctor.org/docs/user-manual/#ordered-lists)
-| Checklists | Checked or unchecked boxes |
-Checked:
-----
-- [*]
-----
-Unchecked:
-----
-- [ ]
-----
-| Multiple levels of lists | bulleted or numbered or combo |
-----
-. Numbered (1), at top level
-* Bullet (2), nested under 1
-* Bullet (3), nested under 1
-. Numbered (4), at top level
-* Bullet (5), nested under 4
-** Bullet (6), nested under 5
-- [x] Checked (7), at top level
-----
-| Labelled lists / variablelists | a list item title or summary followed by content |
-----
-Title:: content
-
-Title::
- content
-----
-|GUI menu cascades | bold text with arrows to show levels | Use an ASCII arrow.
-........
-*File -> Print*
-........
-renders like *File -> Print*
-| Sidebars, quotes, or other blocks of text | a block of text, formatted differently from the default | Delimited using different delimiters, see http://asciidoctor.org/docs/user-manual/#built-in-blocks-summary. Some of the examples above use delimiters like \...., ----,====.
-........
-[example]
-====
-This is an example block.
-====
-
-[source]
-----
-This is a source block.
-----
-
-[note]
-====
-This is a note block.
-====
-
-[quote]
-____
-This is a quote block.
-____
-........
-
-*If you want to insert literal Asciidoc content that keeps being interpreted, when in doubt, use eight dots as the delimiter at the top and bottom.*
-| Nested Sections | chapter, section, sub-section, etc |
-----
-= Book (or chapter if the chapter can be built alone, see leveloffset below)
-
-== Chapter (or section if the chapter is standalone)
-
-=== Section (or subsection, etc)
-
-==== Subsection
-----
-
-and so on up to 6 levels (think carefully about going deeper than 4 levels, maybe you can just titled paragraphs or lists instead). Note that you can include a book inside another book by adding the `:leveloffset:+1` macro directive directly before your include, and resetting it to 0 directly after. See the _book.adoc_ source for examples, as this is how this guide handles chapters. *Don't do it for prefaces, glossaries, appendixes, or other special types of chapters.*
-
-| Include one file from another | Content is included as though it were inline |
-
-----
-include::[/path/to/file.adoc]
-----
-
-For plenty of examples. see _docs/docs.adoc_.
-| A table | a table | See http://asciidoctor.org/docs/user-manual/#tables. Generally rows are separated by newlines and columns by pipes
-| Comment out a single line | A line is skipped during rendering |
-`+//+ This line won't show up`
-| Comment out a block | A section of the file is skipped during rendering |
-----
-////
-Nothing between the slashes will show up.
-////
-----
-| Highlight text for review | text shows up with yellow background |
-----
-Test between #hash marks# is highlighted yellow.
-----
-|===
-
http://git-wip-us.apache.org/repos/asf/kudu/blob/e237fc3a/docs/support/jekyll-templates/document.html.erb
----------------------------------------------------------------------
diff --git a/docs/support/jekyll-templates/document.html.erb b/docs/support/jekyll-templates/document.html.erb
index d7cca20..7d18c04 100644
--- a/docs/support/jekyll-templates/document.html.erb
+++ b/docs/support/jekyll-templates/document.html.erb
@@ -96,7 +96,6 @@ end %>
:schema_design, "Kudu Schema Design",
:transaction_semantics, "Kudu Transaction Semantics",
:contributing, "Contributing to Kudu",
- :style_guide, "Kudu Documentation Style Guide",
:configuration_reference, "Kudu Configuration Reference",
:known_issues, "Known Issues and Limitations",
:export_control, "Export Control Notice"
[4/4] kudu git commit: docs: Add documentation for background tasks
Posted by jd...@apache.org.
docs: Add documentation for background tasks
This patch adds a documentation page for background tasks. My primary
motivation was to document tablet history GC but it seemed that we had a
hole in our documentation concerning the maintenance manager and
background tasks in general, so I tried to document everything at once.
Change-Id: Iaf0911efbb9a5114bb4ae03c1da50335d918eacc
Reviewed-on: http://gerrit.cloudera.org:8080/6501
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Tested-by: Jean-Daniel Cryans <jd...@apache.org>
(cherry picked from commit fcb150c1adcdd4595261433ad81358be7c0cc3a0)
Reviewed-on: http://gerrit.cloudera.org:8080/6548
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/0fada51c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/0fada51c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/0fada51c
Branch: refs/heads/branch-1.3.x
Commit: 0fada51ca26e438544f95b57eb3b7a5ffad51ea1
Parents: e237fc3
Author: Mike Percy <mp...@apache.org>
Authored: Tue Mar 28 12:43:21 2017 -0700
Committer: Jean-Daniel Cryans <jd...@apache.org>
Committed: Tue Apr 4 21:51:35 2017 +0000
----------------------------------------------------------------------
docs/background_tasks.adoc | 103 +++++++++++++++++++
docs/support/jekyll-templates/document.html.erb | 3 +-
2 files changed, 105 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/0fada51c/docs/background_tasks.adoc
----------------------------------------------------------------------
diff --git a/docs/background_tasks.adoc b/docs/background_tasks.adoc
new file mode 100644
index 0000000..48faae1
--- /dev/null
+++ b/docs/background_tasks.adoc
@@ -0,0 +1,103 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+[[background_tasks]]
+= Apache Kudu Background Maintenance Tasks
+:author: Kudu Team
+:imagesdir: ./images
+:icons: font
+:toc: left
+:toclevels: 3
+:doctype: book
+:backend: html5
+:sectlinks:
+:experimental:
+
+Kudu relies on running background tasks for many important automatic
+maintenance activities. These tasks include flushing data from memory to disk,
+compacting data to improve performance, freeing up disk space, and more.
+
+== Maintenance manager
+
+The maintenance manager schedules and runs background tasks. At any given point
+in time, the maintenance manager is prioritizing the next task based on the
+improvement needed at that moment, such as relieving memory pressure, improving
+read performance, or freeing up disk space. The number of worker threads
+dedicated to running background tasks can be controlled by setting
+`--maintenance_manager_num_threads`.
+
+== Flushing data to disk
+
+Flushing data from memory to disk relieves memory pressure and can improve read
+performance by switching from a write-optimized, row-oriented in-memory format
+in the `MemRowSet` to a read-optimized, column-oriented format on disk.
+Background tasks that flush data include `FlushMRSOp` and
+`FlushDeltaMemStoresOp`.
+
+The metrics associated with these ops have the prefix `flush_mrs` and
+`flush_dms`, respectively.
+
+== Compacting on-disk data
+
+Kudu constantly performs several types of compaction tasks in order to maintain
+consistent read and write performance over time. A merging compaction, which combines
+multiple `DiskRowSets` together into a single `DiskRowSet`, is run by
+`CompactRowSetsOp`. There are two types of delta store compaction operations
+that may be run as well: `MinorDeltaCompactionOp` and `MajorDeltaCompactionOp`.
+
+For more information on what these different types of compaction operations do,
+please see the
+link:https://github.com/apache/kudu/blob/master/docs/design-docs/tablet.md[Kudu Tablet
+design document].
+
+The metrics associated with these tasks have the prefix `compact_rs`,
+`delta_minor_compact_rs`, and `delta_major_compact_rs`, respectively.
+
+== Write-ahead log GC
+
+Kudu maintains a write-ahead log (WAL) per tablet that is split into discrete
+fixed-size segments. A tablet periodically rolls the WAL to a new log segment
+when the active segment reaches a configured size (controlled by
+`--log_segment_size_mb`). In order to save disk space and decrease startup
+time, a background task called `LogGCOp` attempts to garbage-collect (GC) old
+WAL segments by deleting them from disk once it is determined that they are no
+longer needed by the local node for durability.
+
+The metrics associated with this background task have the prefix `log_gc`.
+
+== Tablet history GC and the ancient history mark
+
+Because Kudu uses a multiversion concurrency control (MVCC) mechanism to
+ensure that snapshot scans can proceeed isolated from new changes to a table,
+periodically old historical data should be garbage-collected (removed) to free
+up disk space. While Kudu never removes rows or data that are visible in the
+latest version of the data, Kudu does remove records of old changes that are no
+longer visible.
+
+The point in time in the past beyond which historical MVCC data becomes
+inaccessible and is free to be deleted is called the _ancient history mark_
+(AHM). The AHM can be configured by setting `--tablet_history_max_age_sec`.
+
+There are two background tasks that GC historical MVCC data older than the AHM:
+the one that runs the merging compaction, called `CompactRowSetsOp` (see
+above), and a separate background task that deletes old undo delta blocks,
+called `UndoDeltaBlockGCOp`. Running `UndoDeltaBlockGCOp` reduces disk space
+usage in all workloads, but particularly in those with a higher volume of
+updates or upserts.
+
+The metrics associated with this background task have the prefix
+`undo_delta_block`.
http://git-wip-us.apache.org/repos/asf/kudu/blob/0fada51c/docs/support/jekyll-templates/document.html.erb
----------------------------------------------------------------------
diff --git a/docs/support/jekyll-templates/document.html.erb b/docs/support/jekyll-templates/document.html.erb
index 7d18c04..ea3aed0 100644
--- a/docs/support/jekyll-templates/document.html.erb
+++ b/docs/support/jekyll-templates/document.html.erb
@@ -95,9 +95,10 @@ end %>
:developing, "Developing Applications with Kudu",
:schema_design, "Kudu Schema Design",
:transaction_semantics, "Kudu Transaction Semantics",
- :contributing, "Contributing to Kudu",
+ :background_tasks, "Background Maintenance Tasks",
:configuration_reference, "Kudu Configuration Reference",
:known_issues, "Known Issues and Limitations",
+ :contributing, "Contributing to Kudu",
:export_control, "Export Control Notice"
]
toplevels.each_slice(2) do |page, pagename|