You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@accumulo.apache.org by mw...@apache.org on 2017/12/13 16:47:17 UTC
[accumulo-website] branch asf-site updated: Jekyll build from
master:8830c20
This is an automated email from the ASF dual-hosted git repository.
mwalch pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 466ad78 Jekyll build from master:8830c20
466ad78 is described below
commit 466ad7843ff8897fc8cc26be177f5efc79bf9aee
Author: Mike Walch <mw...@apache.org>
AuthorDate: Wed Dec 13 11:42:31 2017 -0500
Jekyll build from master:8830c20
Created post announcing Accumulo tour (#53)
---
.../2017/12/12/take-the-accumulo-tour.html | 83 +++-----
feed.xml | 210 ++++-----------------
index.html | 14 +-
news/index.html | 7 +
tour/index.html | 2 +-
5 files changed, 76 insertions(+), 240 deletions(-)
diff --git a/tour/index.html b/blog/2017/12/12/take-the-accumulo-tour.html
similarity index 79%
copy from tour/index.html
copy to blog/2017/12/12/take-the-accumulo-tour.html
index 81c3190..2696dc4 100644
--- a/tour/index.html
+++ b/blog/2017/12/12/take-the-accumulo-tour.html
@@ -25,7 +25,7 @@
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/bs/jq-2.2.3/dt-1.10.12/datatables.min.css">
<link href="/css/accumulo.css" rel="stylesheet" type="text/css">
-<title>Accumulo Tour</title>
+<title>Take the Accumulo Tour!</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
@@ -146,63 +146,36 @@
</div>
<div id="content">
- <article class="post">
-
- <header class="post-header">
- <h1 class="post-title">Accumulo Tour</h1>
- </header>
-
- <div class="post-content">
-
-<p>Welcome to the Accumulo tour! The tour offers a hands on introduction to the Accumulo Java API, broken down into
-independent steps and exercises. The exercises give you a chance to apply what you have learned by writing code on your
-own. The answers to an exercise are typically provided in the next step. The tour starts with a
-<a href="/tour/getting-started/">Getting Started</a> page that will help you get set up.</p>
-
-<p>When on a tour page, the left and right keys on the keyboard can be used to navigate. If you have any questions
-or suggestions while going through the tour, please send an email to our <a href="/mailing_list/">mailing list</a>
-or <a href="https://issues.apache.org/jira/projects/ACCUMULO">create an issue</a>.</p>
-
-<ol>
- <li>
- <p><a href="/tour/getting-started/">Getting Started</a></p>
- </li>
- <li>
- <p><a href="/tour/basic-read-write/">Writing and Reading</a></p>
- </li>
- <li>
- <p><a href="/tour/data-model/">Data Model</a></p>
- </li>
- <li>
- <p><a href="/tour/data-model-code/">Data Model Code</a></p>
- </li>
- <li>
- <p><a href="/tour/authorizations/">Authorizations</a></p>
- </li>
- <li>
- <p><a href="/tour/authorizations-code/">Authorizations Code</a></p>
- </li>
- <li>
- <p><a href="/tour/ranges-splits/">Ranges and Splits</a></p>
- </li>
- <li>
- <p><a href="/tour/batch-scanner/">Batch Scanner</a></p>
- </li>
- <li>
- <p><a href="/tour/batch-scanner-code/">Batch Scanner Code</a></p>
- </li>
- <li>
- <p><a href="/tour/conditional-writer/">Conditional Writer</a></p>
- </li>
- <li>
- <p><a href="/tour/conditional-writer-code/">Conditional Writer Code</a></p>
- </li>
-</ol>
+ <h1 class="title">Take the Accumulo Tour!</h1>
+
+ <p>
+
+<b>Date: </b> 12 Dec 2017<br>
+
+</p>
+
+<p>Apache Accumulo now has a <a href="/tour/">hands-on tour</a> that introduces users to key Accumulo concepts
+by completing a series of programming exercises. Don’t worry! Solutions are provided if
+you get stuck.</p>
+
+<p>The tour has a <a href="/tour/getting-started/">getting started</a> page that helps you set up your development environment and
+clone a repository with template code. After you are set up, the tour has exercises that cover the
+following Accumulo concepts:</p>
+
+<ul>
+ <li><a href="/tour/basic-read-write/">Writing and Reading from Accumulo</a></li>
+ <li><a href="/tour/data-model/">Data Model</a></li>
+ <li><a href="/tour/authorizations/">Authorizations</a></li>
+ <li><a href="/tour/ranges-splits/">Ranges & Splits</a></li>
+ <li><a href="/tour/batch-scanner/">Batch Scanner</a></li>
+ <li><a href="/tour/conditional-writer/">Conditional Writer</a></li>
+</ul>
+
+<p>More exercises (such as one on Accumulo Iterators) will be added in the future so check for updates.</p>
- </div>
-</article>
+<p><strong>View all posts in the <a href="/news">news archive</a></strong></p>
</div>
diff --git a/feed.xml b/feed.xml
index d13695d..71fb826 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,12 +6,43 @@
</description>
<link>https://accumulo.apache.org/</link>
<atom:link href="https://accumulo.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Tue, 12 Dec 2017 15:56:06 -0500</pubDate>
- <lastBuildDate>Tue, 12 Dec 2017 15:56:06 -0500</lastBuildDate>
+ <pubDate>Wed, 13 Dec 2017 11:42:17 -0500</pubDate>
+ <lastBuildDate>Wed, 13 Dec 2017 11:42:17 -0500</lastBuildDate>
<generator>Jekyll v3.5.2</generator>
<item>
+ <title>Take the Accumulo Tour!</title>
+ <description><p>Apache Accumulo now has a <a href="/tour/">hands-on tour</a> that introduces users to key Accumulo concepts
+by completing a series of programming exercises. Don’t worry! Solutions are provided if
+you get stuck.</p>
+
+<p>The tour has a <a href="/tour/getting-started/">getting started</a> page that helps you set up your development environment and
+clone a repository with template code. After you are set up, the tour has exercises that cover the
+following Accumulo concepts:</p>
+
+<ul>
+ <li><a href="/tour/basic-read-write/">Writing and Reading from Accumulo</a></li>
+ <li><a href="/tour/data-model/">Data Model</a></li>
+ <li><a href="/tour/authorizations/">Authorizations</a></li>
+ <li><a href="/tour/ranges-splits/">Ranges &amp; Splits</a></li>
+ <li><a href="/tour/batch-scanner/">Batch Scanner</a></li>
+ <li><a href="/tour/conditional-writer/">Conditional Writer</a></li>
+</ul>
+
+<p>More exercises (such as one on Accumulo Iterators) will be added in the future so check for updates.</p>
+
+</description>
+ <pubDate>Tue, 12 Dec 2017 00:00:00 -0500</pubDate>
+ <link>https://accumulo.apache.org/blog/2017/12/12/take-the-accumulo-tour.html</link>
+ <guid isPermaLink="true">https://accumulo.apache.org/blog/2017/12/12/take-the-accumulo-tour.html</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Accumulo Summit is on October 16th!</title>
<description><p><a href="http://accumulosummit.com/"><img src="/images/accumulo-summit-2017-logo-horizontal.png" alt="Accumulo Summit" width="400px" /></a></p>
@@ -1350,180 +1381,5 @@ Commands:
</item>
- <item>
- <title>Durability Performance Implications</title>
- <description><h2 id="overview">Overview</h2>
-
-<p>Accumulo stores recently written data in a sorted in memory map. Before data is
-added to this map, it’s written to an unsorted write ahead log(WAL). In the
-case when a tablet server dies, the recently written data is recovered from the
-WAL.</p>
-
-<p>When data is written to Accumulo the following happens :</p>
-
-<ul>
- <li>Client sends a batch of mutations to a tablet server</li>
- <li>Tablet server does the following :
- <ul>
- <li>Writes mutation to tablet servers’ WAL</li>
- <li>Sync or flush tablet servers’ WAL</li>
- <li>Adds mutations to sorted in memory map of each tablet.</li>
- <li>Reports success back to client.</li>
- </ul>
- </li>
-</ul>
-
-<p>The sync/flush step above moves data written to the WAL from memory to disk.
-Write ahead logs are stored in HDFS. HDFS supports two ways of forcing data to
-disk for an open file : <code class="highlighter-rouge">hsync</code> and <code class="highlighter-rouge">hflush</code>.</p>
-
-<h2 id="hdfs-syncflush-details">HDFS Sync/Flush Details</h2>
-
-<p>When <code class="highlighter-rouge">hflush</code> is called on a WAL, it does not guarantee data is on disk. It
-only guarantees that data is in OS buffers on each datanode and on its way to disk.
-As a result calls to <code class="highlighter-rouge">hflush</code> are very fast. If a WAL is replicated to 3 data
-nodes then data may be lost if all three machines reboot or die. If the datanode
-process dies, then data loss will not happen because the data was in OS buffers
-waiting to be written to disk. The machines have to reboot or die for data loss to
-occur.</p>
-
-<p>In order to avoid data loss in the event of reboot, <code class="highlighter-rouge">hsync</code> can be called. This
-will ensure data is written to disk on all datanodes before returning. When
-using <code class="highlighter-rouge">hsync</code> for the WAL, if Accumulo reports success to a user it means the
-data is on disk. However <code class="highlighter-rouge">hsync</code> is much slower than <code class="highlighter-rouge">hflush</code> and the way it’s
-implemented exacerbates the problem. For example <code class="highlighter-rouge">hflush</code> make take 1ms and
-<code class="highlighter-rouge">hsync</code> may take 50ms. This difference will impact writes to Accumulo and can
-be mitigated in some situations with larger buffers in Accumulo.</p>
-
-<p>HDFS keeps checksum data internally by default. Datanodes store checksum data
-in a separate file in the local filesystem. This means when <code class="highlighter-rouge">hsync</code> is called
-on a WAL, two files must be synced on each datanode. Syncing two files doubles
-the time. To make matters even worse, when the two files are synced the local
-filesystem metadata is also synced. Depending on the local filesystem and its
-configuration, syncing the metadata may or may not take time. In the worst
-case, we need to wait for four sync operations at the local filesystem level on
-each datanode. One thing I am not sure about, is if these sync operations occur
-in parallel on the replicas on different datanodes. If anyone can answer this
-question, please let us know on the <a href="/mailing_list">dev list</a>. The following pointers show
-where sync occurs in the datanode code.</p>
-
-<ul>
- <li><a href="https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L358">BlockReceiver.flushOrSync()</a> calls <a href="https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaOutputStreams.java#L78">ReplicaOutputStreams.syncDataOut()</a> a [...]
- <li>The methods in ReplicaOutputStreams call <a href="https://docs.oracle.com/javase/8/docs/api/java/nio/channels/FileChannel.html#force-boolean-">FileChannel.force(true)</a> which
-synchronously flushes data and filesystem metadata.</li>
-</ul>
-
-<p>If files were preallocated (this would avoid syncing local filesystem metadata)
-and checksums were stored in-line, then 1 sync could be done instead of 4.</p>
-
-<h2 id="configuring-wal-flushsync-in-accumulo-16">Configuring WAL flush/sync in Accumulo 1.6</h2>
-
-<p>Accumulo 1.6.0 only supported <code class="highlighter-rouge">hsync</code> and this caused <a href="/release/accumulo-1.6.0#slower-writes-than-previous-accumulo-versions">performance
-problems</a>. In order to offer better performance, the option to
-configure <code class="highlighter-rouge">hflush</code> was <a href="/release/accumulo-1.6.1#write-ahead-log-sync-implementation">added in 1.6.1</a>. The
-<a href="/1.6/accumulo_user_manual#_tserver_wal_sync_method">tserver.wal.sync.method</a> configuration option was added to support
-this feature. This was a tablet server wide option that applied to everything
-written to any table.</p>
-
-<h2 id="group-commit">Group Commit</h2>
-
-<p>Each Accumulo tablet server has a single WAL. When multiple clients send
-mutations to a tablet server at around the same time, the tablet sever may group
-all of this into a single WAL operation. It will do this instead of writing and
-syncing or flushing each client’s mutations to the WAL separately. Doing this
-increase throughput and lowers average latency for clients.</p>
-
-<h2 id="configuring-wal-flushsync-in-accumulo-17">Configuring WAL flush/sync in Accumulo 1.7+</h2>
-
-<p>Accumulo 1.7.0 introduced <a href="/1.7/accumulo_user_manual#_table_durability">table.durability</a>, a new per table property
-for configuring durability. It also stopped using the <code class="highlighter-rouge">tserver.wal.sync.method</code>
-property. The <code class="highlighter-rouge">table.durability</code> property has the following four legal values.
-This property defaults to the most durable option which is <code class="highlighter-rouge">sync</code>.</p>
-
-<ul>
- <li><strong>none</strong> : Do not write to WAL</li>
- <li><strong>log</strong> : Write to WAL, but do not sync</li>
- <li><strong>flush</strong> : Write to WAL and call <code class="highlighter-rouge">hflush</code></li>
- <li><strong>sync</strong> : Write to WAL and call <code class="highlighter-rouge">hsync</code></li>
-</ul>
-
-<p>If multiple writes arrive at around the same time with different durability
-settings, then the group commit code will choose the most durable. This can
-cause one tables settings to slow down writes to another table. Basically, one
-table that is set to <code class="highlighter-rouge">sync</code> can impact the entire system.</p>
-
-<p>In Accumulo 1.6, it was easy to make all writes use <code class="highlighter-rouge">hflush</code> because there was
-only one tserver setting. Getting everything to use <code class="highlighter-rouge">flush</code> in 1.7 and later
-can be a little tricky because by default the Accumulo metadata table is set to
-use <code class="highlighter-rouge">sync</code>. The following shell commands show this. The first command sets
-<code class="highlighter-rouge">table.durability=flush</code> as a system wide default for all tables. However, the
-metadata table is still set to <code class="highlighter-rouge">sync</code>, because it has a per table override for
-that setting. This override is set when Accumulo is initialized. To get this
-table to use <code class="highlighter-rouge">flush</code>, the per table override must be deleted. After deleting
-those properties, the metadata tables will inherit the system wide setting.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>root@uno&gt; config -s table.durability=flush
-root@uno&gt; createtable foo
-root@uno foo&gt; config -t foo -f table.durability
------------+---------------------+----------------------------------------------
-SCOPE | NAME | VALUE
------------+---------------------+----------------------------------------------
-default | table.durability .. | sync
-system | @override ...... | flush
------------+---------------------+----------------------------------------------
-root@uno&gt; config -t accumulo.metadata -f table.durability
------------+---------------------+----------------------------------------------
-SCOPE | NAME | VALUE
------------+---------------------+----------------------------------------------
-default | table.durability .. | sync
-system | @override ...... | flush
-table | @override ...... | sync
------------+---------------------+----------------------------------------------
-root@uno&gt; config -t accumulo.metadata -d table.durability
-root@uno&gt; config -t accumulo.metadata -f table.durability
------------+---------------------+----------------------------------------------
-SCOPE | NAME | VALUE
------------+---------------------+----------------------------------------------
-default | table.durability .. | sync
-system | @override ...... | flush
------------+---------------------+----------------------------------------------
-</code></pre>
-</div>
-
-<p>In short, executing the following commands will make all writes use <code class="highlighter-rouge">flush</code>
-(assuming no other tables or namespaces have been specifically set to <code class="highlighter-rouge">sync</code>).</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>config -s table.durability=flush
-config -t accumulo.metadata -d table.durability
-config -t accumulo.root -d table.durability
-</code></pre>
-</div>
-
-<p>Even with these settings adjusted, minor compactions could still force <code class="highlighter-rouge">hsync</code>
-to be called in 1.7.0 and 1.7.1. This was fixed in 1.7.2 and 1.8.0. See the
-<a href="/release/accumulo-1.7.2#minor-performance-improvements">1.7.2 release notes</a> and <a href="https://issues.apache.org/jira/browse/ACCUMULO-4112">ACCUMULO-4112</a> for more details.</p>
-
-<p>In addition to the per table durability setting, a per batch writer durability
-setting was also added in 1.7.0. See
-<a href="/1.8/apidocs/org/apache/accumulo/core/client/BatchWriterConfig.html#setDurability(org.apache.accumulo.core.client.Durability)">BatchWriterConfig.setDurability(…)</a>. This means any client could
-potentially cause a <code class="highlighter-rouge">hsync</code> operation to occur, even if the system is
-configured to use <code class="highlighter-rouge">hflush</code>.</p>
-
-<h2 id="improving-the-situation">Improving the situation</h2>
-
-<p>The more granular durability settings introduced in 1.7.0 can cause some
-unexpected problems. <a href="https://issues.apache.org/jira/browse/ACCUMULO-4146">ACCUMULO-4146</a> suggests one possible way to solve these
-problems with Per-durability write ahead logs.</p>
-
-</description>
- <pubDate>Wed, 02 Nov 2016 13:00:00 -0400</pubDate>
- <link>https://accumulo.apache.org/blog/2016/11/02/durability-performance.html</link>
- <guid isPermaLink="true">https://accumulo.apache.org/blog/2016/11/02/durability-performance.html</guid>
-
-
- <category>blog</category>
-
- </item>
-
</channel>
</rss>
diff --git a/index.html b/index.html
index a6403e8..89eef50 100644
--- a/index.html
+++ b/index.html
@@ -186,6 +186,13 @@
<div class="row latest-news-item">
<div class="col-sm-12" style="margin-bottom: 5px">
+ <span style="font-size: 12px; margin-right: 5px;">Dec 2017</span>
+ <a href="/blog/2017/12/12/take-the-accumulo-tour.html">Take the Accumulo Tour!</a>
+ </div>
+ </div>
+
+ <div class="row latest-news-item">
+ <div class="col-sm-12" style="margin-bottom: 5px">
<span style="font-size: 12px; margin-right: 5px;">Jul 2017</span>
<a href="/blog/2017/07/27/accumulo-summmit-on-october-16.html">Accumulo Summit is on October 16th!</a>
</div>
@@ -212,13 +219,6 @@
</div>
</div>
- <div class="row latest-news-item">
- <div class="col-sm-12" style="margin-bottom: 5px">
- <span style="font-size: 12px; margin-right: 5px;">Mar 2017</span>
- <a href="/blog/2017/03/21/happy-anniversary-accumulo.html">Happy Anniversary Accumulo</a>
- </div>
- </div>
-
<div id="news-archive-link">
<p>View all posts in the <a href="/news">news archive</a></p>
</div>
diff --git a/news/index.html b/news/index.html
index b2430ae..447aaa7 100644
--- a/news/index.html
+++ b/news/index.html
@@ -156,6 +156,13 @@
<div class="row" style="margin-top: 15px">
+ <div class="col-md-1">Dec 12</div>
+ <div class="col-md-10"><a href="/blog/2017/12/12/take-the-accumulo-tour.html">Take the Accumulo Tour!</a></div>
+ </div>
+
+
+
+ <div class="row" style="margin-top: 15px">
<div class="col-md-1">Jul 27</div>
<div class="col-md-10"><a href="/blog/2017/07/27/accumulo-summmit-on-october-16.html">Accumulo Summit is on October 16th!</a></div>
</div>
diff --git a/tour/index.html b/tour/index.html
index 81c3190..e0cd0d3 100644
--- a/tour/index.html
+++ b/tour/index.html
@@ -160,7 +160,7 @@ own. The answers to an exercise are typically provided in the next step. The to
<a href="/tour/getting-started/">Getting Started</a> page that will help you get set up.</p>
<p>When on a tour page, the left and right keys on the keyboard can be used to navigate. If you have any questions
-or suggestions while going through the tour, please send an email to our <a href="/mailing_list/">mailing list</a>
+or suggestions while going through the tour, please send an email to our <a href="/contact-us/#mailing-lists">mailing list</a>
or <a href="https://issues.apache.org/jira/projects/ACCUMULO">create an issue</a>.</p>
<ol>
--
To stop receiving notification emails like this one, please contact
['"commits@accumulo.apache.org" <co...@accumulo.apache.org>'].