You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by mp...@apache.org on 2018/09/26 17:56:36 UTC
[1/4] kudu-site git commit: Publish commit(s) from site source repo:
83530755d Blogpost describing index skip scan optimization.
Repository: kudu-site
Updated Branches:
refs/heads/asf-site 8efb67b1a -> 12782cec3
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/img/index-skip-scan/skip-scan-example-table.png
----------------------------------------------------------------------
diff --git a/img/index-skip-scan/skip-scan-example-table.png b/img/index-skip-scan/skip-scan-example-table.png
new file mode 100644
index 0000000..3b965ac
Binary files /dev/null and b/img/index-skip-scan/skip-scan-example-table.png differ
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/img/index-skip-scan/skip-scan-performance-graph.png
----------------------------------------------------------------------
diff --git a/img/index-skip-scan/skip-scan-performance-graph.png b/img/index-skip-scan/skip-scan-performance-graph.png
new file mode 100644
index 0000000..5004b65
Binary files /dev/null and b/img/index-skip-scan/skip-scan-performance-graph.png differ
[3/4] kudu-site git commit: Publish commit(s) from site source repo:
83530755d Blogpost describing index skip scan optimization.
Posted by mp...@apache.org.
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/08/31/intro-flume-kudu-sink.html
----------------------------------------------------------------------
diff --git a/2016/08/31/intro-flume-kudu-sink.html b/2016/08/31/intro-flume-kudu-sink.html
index 4bf4f7d..7f94bc9 100644
--- a/2016/08/31/intro-flume-kudu-sink.html
+++ b/2016/08/31/intro-flume-kudu-sink.html
@@ -225,7 +225,7 @@ release and the source code can be found <a href="https://github.com/apache/kudu
<p>Here is a sample flume configuration file:</p>
-<pre><code>agent1.sources = source1
+<div class="highlighter-rouge">agent1.sources = source1
agent1.channels = channel1
agent1.sinks = sink1
@@ -243,25 +243,25 @@ agent1.sinks.sink1.tableName = stats
agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.batchSize = 50
agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-</code></pre>
+</div>
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
+<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using
+<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is
the built-in and default producer, but it’s implemented as a showcase for how to write Flume
events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
+<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
table, it has to be created before the Kudu Flume Sink is started.</p>
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
+<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
impact on ingest performance of the Kudu cluster.</p>
@@ -311,89 +311,89 @@ impact on ingest performance of the Kudu cluster.</p>
<p>Let’s take a look at the source code for the built-in producer class:</p>
-<pre><code class="language-java">public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;
-
- public SimpleKuduEventProducer(){
- }
-
- @Override
- public void configure(Context context) {
- payloadColumn = context.getString("payloadColumn","payload");
- }
-
- @Override
- public void configure(ComponentConfiguration conf) {
- }
-
- @Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }
-
- @Override
- public List<Operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);
-
- return Collections.singletonList((Operation) insert);
- } catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
- }
- }
-
- @Override
- public void close() {
- }
-}
-</code></pre>
-
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+<div class="highlighter-rouge"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
+
+ <span class="kd">public</span> <span class="nf">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
+
+ <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="nf">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</div>
+
+<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
which itself looks like this:</p>
-<pre><code class="language-java">public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
+<div class="highlighter-rouge"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
* Initialize the event producer.
* @param event to be written to Kudu
* @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);
+ */</span>
+ <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span>
- /**
+ <span class="cm">/**
* Get the operations that should be written out to Kudu as a result of this
* event. This list is written to Kudu using the Kudu client API.
* @return List of {@link org.kududb.client.Operation} which
* are written as such to Kudu
- */
- List<Operation> getOperations();
+ */</span>
+ <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">();</span>
- /*
+ <span class="cm">/*
* Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-</code></pre>
+ */</span>
+ <span class="kt">void</span> <span class="nf">close</span><span class="o">();</span>
+<span class="o">}</span>
+</div>
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
+<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+<code class="highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
+configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table.
+<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
example).</p>
-<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In
+<p>The main producer logic resides in the <code class="highlighter-rouge">public List<Operation> getOperations()</code> method. In
SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
+the Kudu table. Here we call Kudu’s <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
will need to write your own custom producer in the real world, but you can base your implementation
on the built-in ones.</p>
@@ -423,6 +423,8 @@ is included in the Kudu distribution. You can follow him on Twitter at
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -451,8 +453,6 @@ is included in the Kudu distribution. You can follow him on Twitter at
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/09/16/predicate-pushdown.html
----------------------------------------------------------------------
diff --git a/2016/09/16/predicate-pushdown.html b/2016/09/16/predicate-pushdown.html
index 8af7be3..a1c26a2 100644
--- a/2016/09/16/predicate-pushdown.html
+++ b/2016/09/16/predicate-pushdown.html
@@ -269,6 +269,8 @@ coordinators, and from the Cloudera community as a whole.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -297,8 +299,6 @@ coordinators, and from the Cloudera community as a whole.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/09/20/apache-kudu-1-0-0-released.html
----------------------------------------------------------------------
diff --git a/2016/09/20/apache-kudu-1-0-0-released.html b/2016/09/20/apache-kudu-1-0-0-released.html
index 5c740fd..c8deda1 100644
--- a/2016/09/20/apache-kudu-1-0-0-released.html
+++ b/2016/09/20/apache-kudu-1-0-0-released.html
@@ -132,15 +132,15 @@ history of all changes made to a given table since the beginning of time.</p>
</li>
<li>
<p>Most of Kudu’s command line tools have been consolidated under a new
-top-level <code>kudu</code> tool. This reduces the number of large binaries distributed
+top-level <code class="highlighter-rouge">kudu</code> tool. This reduces the number of large binaries distributed
with Kudu and also includes much-improved help output.</p>
</li>
<li>
- <p>Administrative tools including <code>kudu cluster ksck</code> now support running
+ <p>Administrative tools including <code class="highlighter-rouge">kudu cluster ksck</code> now support running
against multi-master Kudu clusters.</p>
</li>
<li>
- <p>The C++ client API now supports writing data in <code>AUTO_FLUSH_BACKGROUND</code> mode.
+ <p>The C++ client API now supports writing data in <code class="highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> mode.
This can provide higher throughput for ingest workloads.</p>
</li>
</ul>
@@ -164,6 +164,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -192,8 +194,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/09/26/strata-nyc-kudu-talks.html
----------------------------------------------------------------------
diff --git a/2016/09/26/strata-nyc-kudu-talks.html b/2016/09/26/strata-nyc-kudu-talks.html
index 27e8926..e3f79fb 100644
--- a/2016/09/26/strata-nyc-kudu-talks.html
+++ b/2016/09/26/strata-nyc-kudu-talks.html
@@ -183,6 +183,8 @@ Be sure to RSVP as spots are filling up fast.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -211,8 +213,6 @@ Be sure to RSVP as spots are filling up fast.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/10/11/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/10/11/weekly-update.html b/2016/10/11/weekly-update.html
index 3ce1988..03d8a34 100644
--- a/2016/10/11/weekly-update.html
+++ b/2016/10/11/weekly-update.html
@@ -189,14 +189,14 @@ look and leave your comments! Similarly, if you are interested in contributing
in any of these areas, please feel free to volunteer on the mailing list.
Help of all kinds (coding, documentation, testing, etc) is welcomed.</p>
</li>
- <li>Adar Dembo spent a chunk of time re-working the <code>thirdparty</code> directory
+ <li>Adar Dembo spent a chunk of time re-working the <code class="highlighter-rouge">thirdparty</code> directory
that contains most of Kudu’s native dependencies. The major resulting
changes are:
<ul>
<li>Build directories are now cleanly isolated from source directories,
improving cleanliness of re-builds.</li>
- <li>ThreadSanitizer (TSAN) builds now use <code>libc++</code> instead of <code>libstdcxx</code>
-for C++ library support. The <code>libc++</code> library has better support for
+ <li>ThreadSanitizer (TSAN) builds now use <code class="highlighter-rouge">libc++</code> instead of <code class="highlighter-rouge">libstdcxx</code>
+for C++ library support. The <code class="highlighter-rouge">libc++</code> library has better support for
sanitizers, is easier to build in isolation, and solves some compatibility
issues that Adar was facing with GCC 5 on Ubuntu Xenial.</li>
<li>All of the thirdparty dependencies now build with TSAN instrumentation,
@@ -236,14 +236,14 @@ to increase.</p>
<li>
<p>Dan Burkert picked up work originally started by Sameer Abhyankar on
<a href="https://issues.apache.org/jira/browse/KUDU-1363">KUDU-1363</a>, which adds
-support for adding <code>IN (...)</code> predicates to scanners. Dan committed the
+support for adding <code class="highlighter-rouge">IN (...)</code> predicates to scanners. Dan committed the
<a href="http://gerrit.cloudera.org:8080/2986">main patch</a> as well as corresponding
<a href="http://gerrit.cloudera.org:8080/4530">support in the Java client</a>.
Jordan Birdsell quickly added corresponding support in <a href="http://gerrit.cloudera.org:8080/4548">Python</a>.
This new feature will be available in an upcoming release.</p>
</li>
<li>
- <p>Work continues on the <code>kudu</code> command line tool. Dinesh Bhat added
+ <p>Work continues on the <code class="highlighter-rouge">kudu</code> command line tool. Dinesh Bhat added
the ability to ask a tablet’s leader to <a href="http://gerrit.cloudera.org:8080/4533">step down</a>
and Alexey Serbin added a <a href="http://gerrit.cloudera.org:8080/4412">tool to insert random data into a
table</a>.</p>
@@ -287,6 +287,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -315,8 +317,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/10/20/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/10/20/weekly-update.html b/2016/10/20/weekly-update.html
index 90f273d..cb641b8 100644
--- a/2016/10/20/weekly-update.html
+++ b/2016/10/20/weekly-update.html
@@ -216,6 +216,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -244,8 +246,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/11/01/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/11/01/weekly-update.html b/2016/11/01/weekly-update.html
index 7990409..66ee4ae 100644
--- a/2016/11/01/weekly-update.html
+++ b/2016/11/01/weekly-update.html
@@ -214,6 +214,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -242,8 +244,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/11/15/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/11/15/weekly-update.html b/2016/11/15/weekly-update.html
index 3ad0360..03ae6a3 100644
--- a/2016/11/15/weekly-update.html
+++ b/2016/11/15/weekly-update.html
@@ -235,6 +235,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -263,8 +265,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/01/20/apache-kudu-1-2-0-released.html
----------------------------------------------------------------------
diff --git a/2017/01/20/apache-kudu-1-2-0-released.html b/2017/01/20/apache-kudu-1-2-0-released.html
index 104074f..aa635bb 100644
--- a/2017/01/20/apache-kudu-1-2-0-released.html
+++ b/2017/01/20/apache-kudu-1-2-0-released.html
@@ -160,6 +160,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -188,8 +190,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/03/20/apache-kudu-1-3-0-released.html
----------------------------------------------------------------------
diff --git a/2017/03/20/apache-kudu-1-3-0-released.html b/2017/03/20/apache-kudu-1-3-0-released.html
index 4dc68e5..5ba47db 100644
--- a/2017/03/20/apache-kudu-1-3-0-released.html
+++ b/2017/03/20/apache-kudu-1-3-0-released.html
@@ -159,6 +159,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -187,8 +189,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/04/19/apache-kudu-1-3-1-released.html
----------------------------------------------------------------------
diff --git a/2017/04/19/apache-kudu-1-3-1-released.html b/2017/04/19/apache-kudu-1-3-1-released.html
index fcd206f..4482d7a 100644
--- a/2017/04/19/apache-kudu-1-3-1-released.html
+++ b/2017/04/19/apache-kudu-1-3-1-released.html
@@ -143,6 +143,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -171,8 +173,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/06/13/apache-kudu-1-4-0-released.html
----------------------------------------------------------------------
diff --git a/2017/06/13/apache-kudu-1-4-0-released.html b/2017/06/13/apache-kudu-1-4-0-released.html
index b484adb..6e1a205 100644
--- a/2017/06/13/apache-kudu-1-4-0-released.html
+++ b/2017/06/13/apache-kudu-1-4-0-released.html
@@ -132,7 +132,7 @@ improvements, optimizations, and bug fixes.</p>
<li>a new C++ client API to efficiently map primary keys to their associated partitions
and hosts</li>
<li>support for long-running fault-tolerant scans in the Java client</li>
- <li>a new <code>kudu fs check</code> command which can perform offline consistency checks
+ <li>a new <code class="highlighter-rouge">kudu fs check</code> command which can perform offline consistency checks
and repairs on the local on-disk storage of a Tablet Server or Master.</li>
<li>many optimizations to reduce disk space usage, improve write throughput,
and improve throughput of background maintenance operations.</li>
@@ -159,6 +159,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -187,8 +189,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/09/08/apache-kudu-1-5-0-released.html
----------------------------------------------------------------------
diff --git a/2017/09/08/apache-kudu-1-5-0-released.html b/2017/09/08/apache-kudu-1-5-0-released.html
index dccee96..df182a4 100644
--- a/2017/09/08/apache-kudu-1-5-0-released.html
+++ b/2017/09/08/apache-kudu-1-5-0-released.html
@@ -136,9 +136,9 @@ scenarios</li>
additional reductions planned for the future</li>
<li>a new configuration dashboard on the web UI which provides a high-level
summary of important configuration values</li>
- <li>a new <code>kudu tablet move</code> command which moves a tablet replica from one tablet
+ <li>a new <code class="highlighter-rouge">kudu tablet move</code> command which moves a tablet replica from one tablet
server to another</li>
- <li>a new <code>kudu local_replica data_size</code> command which summarizes the space usage
+ <li>a new <code class="highlighter-rouge">kudu local_replica data_size</code> command which summarizes the space usage
of a local tablet</li>
<li>all on-disk data is now checksummed by default, which provides error detection
for improved confidence when running Kudu on unreliable hardware</li>
@@ -165,6 +165,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -193,8 +195,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/09/18/kudu-consistency-pt1.html
----------------------------------------------------------------------
diff --git a/2017/09/18/kudu-consistency-pt1.html b/2017/09/18/kudu-consistency-pt1.html
index 3f40265..39f6c65 100644
--- a/2017/09/18/kudu-consistency-pt1.html
+++ b/2017/09/18/kudu-consistency-pt1.html
@@ -237,29 +237,29 @@ have increasing timestamps, depending on the user’s choices.</p>
<p>Row mutations performed by a single client <em>instance</em> are guaranteed to have increasing timestamps
thus reflecting their potential causal relationship. This property is always enforced. However
there are two major <em>“knobs”</em> that are available to the user to make performance trade-offs, the
-<code>Read</code> mode, and the <code>External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a>
+<code class="highlighter-rouge">Read</code> mode, and the <code class="highlighter-rouge">External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a>
for more information on how to use the relevant APIs).</p>
-<p>The first and most important knob, the <code>Read</code> mode, pertains to what is the guaranteed recency of
+<p>The first and most important knob, the <code class="highlighter-rouge">Read</code> mode, pertains to what is the guaranteed recency of
data resulting from scans. Since Kudu uses replication for availability and fault-tolerance, there
are always multiple replicas of any data item.
Not all replicas must be up-to-date so if the user cares about recency, e.g. if the user requires
that any data read includes all previously written data <em>from a single client instance</em> then it must
-choose the <code>READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe
+choose the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe
<strong>“READ YOUR OWN WRITES”</strong> semantics, i.e. scans from a client will always include all previous mutations
performed by that client. Note that this property is local to a single client instance, not a global
property.</p>
-<p>The second “knob”, the <code>External Consistency</code> mode, defines the semantics of how reads and writes
-are performed across multiple client instances. By default, <code>External Consistency</code> is set to
- <code>CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even
+<p>The second “knob”, the <code class="highlighter-rouge">External Consistency</code> mode, defines the semantics of how reads and writes
+are performed across multiple client instances. By default, <code class="highlighter-rouge">External Consistency</code> is set to
+ <code class="highlighter-rouge">CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even
across different machines) if they are performing writes/reads that are somehow causally linked.
If done correctly this enables <strong>STRICT SERIALIZABILITY</strong>[5], i.e. <strong>LINEARIZABILITY</strong>[6] and
<strong>SERIALIZABILITY</strong>[7] at the same time, at the cost of having the user coordinate the timestamp
tokens across clients (a survey of the meaning of these, and other definitions can be found
<a href="http://www.ics.forth.gr/tech-reports/2013/2013.TR439_Survey_on_Consistency_Conditions.pdf">here</a>).
-The alternative setting for <code>External Consistency</code> is to have it set to
-<code>COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by
+The alternative setting for <code class="highlighter-rouge">External Consistency</code> is to have it set to
+<code class="highlighter-rouge">COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by
implementing Google Spanner’s <em>TrueTime</em>. This comes at the cost of higher latency (depending on how
tightly synchronized the system clocks of the various tablet servers are), but doesn’t require users
to propagate timestamps programmatically.</p>
@@ -302,6 +302,8 @@ to enable the consistency semantics introduced in the previous section, includin
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -330,8 +332,6 @@ to enable the consistency semantics introduced in the previous section, includin
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/10/23/nosql-kudu-spanner-slides.html
----------------------------------------------------------------------
diff --git a/2017/10/23/nosql-kudu-spanner-slides.html b/2017/10/23/nosql-kudu-spanner-slides.html
index 48e7a43..0deb771 100644
--- a/2017/10/23/nosql-kudu-spanner-slides.html
+++ b/2017/10/23/nosql-kudu-spanner-slides.html
@@ -183,6 +183,8 @@ below:</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -211,8 +213,6 @@ below:</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2017/12/08/apache-kudu-1-6-0-released.html
----------------------------------------------------------------------
diff --git a/2017/12/08/apache-kudu-1-6-0-released.html b/2017/12/08/apache-kudu-1-6-0-released.html
index e82f170..59f6977 100644
--- a/2017/12/08/apache-kudu-1-6-0-released.html
+++ b/2017/12/08/apache-kudu-1-6-0-released.html
@@ -184,6 +184,8 @@ Maven repository and are
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -212,8 +214,6 @@ Maven repository and are
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/03/23/apache-kudu-1-7-0-released.html
----------------------------------------------------------------------
diff --git a/2018/03/23/apache-kudu-1-7-0-released.html b/2018/03/23/apache-kudu-1-7-0-released.html
index b495689..9017af5 100644
--- a/2018/03/23/apache-kudu-1-7-0-released.html
+++ b/2018/03/23/apache-kudu-1-7-0-released.html
@@ -196,6 +196,8 @@ Maven repository and are
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -224,8 +226,6 @@ Maven repository and are
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/07/10/instrumentation-in-kudu.html
----------------------------------------------------------------------
diff --git a/2018/07/10/instrumentation-in-kudu.html b/2018/07/10/instrumentation-in-kudu.html
index f42eeba..f019b04 100644
--- a/2018/07/10/instrumentation-in-kudu.html
+++ b/2018/07/10/instrumentation-in-kudu.html
@@ -156,6 +156,8 @@ below. My talk spans the first 34 minutes.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -184,8 +186,6 @@ below. My talk spans the first 34 minutes.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
----------------------------------------------------------------------
diff --git a/2018/08/06/getting-started-with-kudu-an-oreilly-title.html b/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
index 95f74a1..8e5916f 100644
--- a/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
+++ b/2018/08/06/getting-started-with-kudu-an-oreilly-title.html
@@ -132,9 +132,9 @@ challenge at that time.
In that context, on October 11th 2012 Todd Lipcon perform Apache Kudu’s initial
commit. The commit message was:</p>
-<pre><code>Code for writing cfiles seems to basically work
+<div class="highlighter-rouge">Code for writing cfiles seems to basically work
Need to write code for reading cfiles, still
-</code></pre>
+</div>
<p>And Kudu development was off and running. Around this same time Todd, on his
internal Wiki page, started listing out the papers he was reading to develop
@@ -170,7 +170,7 @@ of Kudu. Specifically you will learn:</p>
<p>Looking forward, I am excited to see Kudu gain additional features and adoption
and eventually the second revision of this title. In the meantime, if you have
-feedback or questions, please reach out on the <code>#getting-started-kudu</code> channel of
+feedback or questions, please reach out on the <code class="highlighter-rouge">#getting-started-kudu</code> channel of
the <a href="https://getkudu-slack.herokuapp.com/">Kudu Slack</a> or if you prefer non-real-time
communication, please use the user@ mailing list!</p>
@@ -183,6 +183,8 @@ communication, please use the user@ mailing list!</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -211,8 +213,6 @@ communication, please use the user@ mailing list!</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/09/11/simplified-pipelines-with-kudu.html
----------------------------------------------------------------------
diff --git a/2018/09/11/simplified-pipelines-with-kudu.html b/2018/09/11/simplified-pipelines-with-kudu.html
index d8c92c1..69c9985 100644
--- a/2018/09/11/simplified-pipelines-with-kudu.html
+++ b/2018/09/11/simplified-pipelines-with-kudu.html
@@ -166,6 +166,8 @@ the backend.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -194,8 +196,6 @@ the backend.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2018/09/26/index-skip-scan-optimization-in-kudu.html
----------------------------------------------------------------------
diff --git a/2018/09/26/index-skip-scan-optimization-in-kudu.html b/2018/09/26/index-skip-scan-optimization-in-kudu.html
new file mode 100644
index 0000000..bf62385
--- /dev/null
+++ b/2018/09/26/index-skip-scan-optimization-in-kudu.html
@@ -0,0 +1,320 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta http-equiv="X-UA-Compatible" content="IE=edge" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" />
+ <meta name="author" content="Cloudera" />
+ <title>Apache Kudu - Index Skip Scan Optimization in Kudu</title>
+ <!-- Bootstrap core CSS -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"
+ integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7"
+ crossorigin="anonymous">
+
+ <!-- Custom styles for this template -->
+ <link href="/css/kudu.css" rel="stylesheet"/>
+ <link href="/css/asciidoc.css" rel="stylesheet"/>
+ <link rel="shortcut icon" href="/img/logo-favicon.ico" />
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css" />
+
+
+ <link rel="alternate" type="application/atom+xml"
+ title="RSS Feed for Apache Kudu blog"
+ href="/feed.xml" />
+
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+ <div class="kudu-site container-fluid">
+ <!-- Static navbar -->
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a class="logo" href="/"><img
+ src="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png"
+ srcset="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png 1x, //d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_160px.png 2x"
+ alt="Apache Kudu"/></a>
+
+ </div>
+ <div id="navbar" class="collapse navbar-collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li >
+ <a href="/">Home</a>
+ </li>
+ <li >
+ <a href="/overview.html">Overview</a>
+ </li>
+ <li >
+ <a href="/docs/">Documentation</a>
+ </li>
+ <li >
+ <a href="/releases/">Download</a>
+ </li>
+ <li class="active">
+ <a href="/blog/">Blog</a>
+ </li>
+ <!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here
+ that doesn't also appear elsewhere on the site. -->
+ <li class="dropdown">
+ <a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li class="dropdown-header">GET IN TOUCH</li>
+ <li><a class="icon email" href="/community.html">Mailing Lists</a></li>
+ <li><a class="icon slack" href="https://getkudu-slack.herokuapp.com/">Slack Channel</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li>
+ <li><a href="/committers.html">Project Committers</a></li>
+ <!--<li><a href="/roadmap.html">Roadmap</a></li>-->
+ <li><a href="/community.html#contributions">How to Contribute</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">DEVELOPER RESOURCES</li>
+ <li><a class="icon github" href="https://github.com/apache/incubator-kudu">GitHub</a></li>
+ <li><a class="icon gerrit" href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu">Gerrit Code Review</a></li>
+ <li><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU">JIRA Issue Tracker</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SOCIAL MEDIA</li>
+ <li><a class="icon twitter" href="https://twitter.com/ApacheKudu">Twitter</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">APACHE SOFTWARE FOUNDATION</li>
+ <li><a href="https://www.apache.org/security/" target="_blank">Security</a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li>
+ <li><a href="https://www.apache.org/licenses/" target="_blank">License</a></li>
+ </ul>
+ </li>
+ <li >
+ <a href="/faq.html">FAQ</a>
+ </li>
+ </ul><!-- /.nav -->
+ </div><!-- /#navbar -->
+ </div><!-- /.container-fluid -->
+ </nav>
+
+<div class="row header">
+ <div class="col-lg-12">
+ <h2><a href="/blog">Apache Kudu Blog</a></h2>
+ </div>
+</div>
+
+<div class="row-fluid">
+ <div class="col-lg-9">
+ <article>
+ <header>
+ <h1 class="entry-title">Index Skip Scan Optimization in Kudu</h1>
+ <p class="meta">Posted 26 Sep 2018 by Anupama Gupta</p>
+ </header>
+ <div class="entry-content">
+ <p>This summer I got the opportunity to intern with the Apache Kudu team at Cloudera.
+My project was to optimize the Kudu scan path by implementing a technique called
+index skip scan (a.k.a. scan-to-seek, see section 4.1 in [1]). I wanted to share
+my experience and the progress we’ve made so far on the approach.</p>
+
+<!--more-->
+
+<p>Let’s begin with discussing the current query flow in Kudu.
+Consider the following table:</p>
+
+<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">metrics</span> <span class="p">(</span>
+ <span class="k">host</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="n">tstamp</span> <span class="nb">INT</span><span class="p">,</span>
+ <span class="n">clusterid</span> <span class="nb">INT</span><span class="p">,</span>
+ <span class="k">role</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">tstamp</span><span class="p">,</span> <span class="n">clusterid</span><span class="p">)</span>
+<span class="p">);</span></code></pre></div>
+
+<p><img src="/img/index-skip-scan/example-table.png" alt="png" class="img-responsive" />
+<em>Sample rows of table <code class="highlighter-rouge">metrics</code> (sorted by key columns).</em></p>
+
+<p>In this case, by default, Kudu internally builds a primary key index (implemented as a
+<a href="https://en.wikipedia.org/wiki/B-tree">B-tree</a>) for the table <code class="highlighter-rouge">metrics</code>.
+As shown in the table above, the index data is sorted by the composite of all key columns.
+When the user query contains the first key column (<code class="highlighter-rouge">host</code>), Kudu uses the index (as the index data is
+primarily sorted on the first key column).</p>
+
+<p>Now, what if the user query does not contain the first key column and instead only contains the <code class="highlighter-rouge">tstamp</code> column?
+In the above case, the <code class="highlighter-rouge">tstamp</code> column values are sorted with respect to <code class="highlighter-rouge">host</code>,
+but are not globally sorted, and as such, it’s non-trivial to use the index to filter rows.
+Instead, a full tablet scan is done by default. Other databases may optimize such scans by building secondary indexes
+(though it might be redundant to build one on one of the primary keys). However, this isn’t an option for Kudu,
+given its lack of secondary index support.</p>
+
+<p>The question is, can Kudu do better than a full tablet scan here?</p>
+
+<p>The answer is yes! Let’s observe the column preceding the <code class="highlighter-rouge">tstamp</code> column. We will refer to it as the
+“prefix column” and its specific value as the “prefix key”. In this example, <code class="highlighter-rouge">host</code> is the prefix column.
+Note that the prefix keys are sorted in the index and that all rows of a given prefix key are also sorted by the
+remaining key columns. Therefore, we can use the index to skip to the rows that have distinct prefix keys,
+and also satisfy the predicate on the <code class="highlighter-rouge">tstamp</code> column.
+For example, consider the query:</p>
+
+<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="n">clusterid</span> <span class="k">FROM</span> <span class="n">metrics</span> <span class="k">WHERE</span> <span class="n">tstamp</span> <span class="o">=</span> <span class="mi">100</span><span class="p">;</span></code></pre></div>
+
+<p><img src="/img/index-skip-scan/skip-scan-example-table.png" alt="png" class="img-responsive" />
+<em>Skip scan flow illustration. The rows in green are scanned and the rest are skipped.</em></p>
+
+<p>The tablet server can use the index to <strong>skip</strong> to the first row with a distinct prefix key (<code class="highlighter-rouge">host = helium</code>) that
+matches the predicate (<code class="highlighter-rouge">tstamp = 100</code>) and then <strong>scan</strong> through the rows until the predicate no longer matches. At that
+point we would know that no more rows with <code class="highlighter-rouge">host = helium</code> will satisfy the predicate, and we can skip to the next
+prefix key. This holds true for all distinct keys of <code class="highlighter-rouge">host</code>. Hence, this method is popularly known as
+<strong>skip scan optimization</strong>[2, 3].</p>
+
+<h1 id="performance">Performance</h1>
+
+<p>This optimization can speed up queries significantly, depending on the cardinality (number of distinct values) of the
+prefix column. The lower the prefix column cardinality, the better the skip scan performance. In fact, when the
+prefix column cardinality is high, skip scan is not a viable approach. The performance graph (obtained using the example
+schema and query pattern mentioned earlier) is shown below.</p>
+
+<p>Based on our experiments, on up to 10 million rows per tablet (as shown below), we found that the skip scan performance
+begins to get worse with respect to the full tablet scan performance when the prefix column cardinality
+exceeds sqrt(number_of_rows_in_tablet).
+Therefore, in order to use skip scan performance benefits when possible and maintain a consistent performance in cases
+of large prefix column cardinality, we have tentatively chosen to dynamically disable skip scan when the number of skips for
+distinct prefix keys exceeds sqrt(number_of_rows_in_tablet).
+It will be an interesting project to further explore sophisticated heuristics to decide when
+to dynamically disable skip scan.</p>
+
+<p><img src="/img/index-skip-scan/skip-scan-performance-graph.png" alt="png" class="img-responsive" /></p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>Skip scan optimization in Kudu can lead to huge performance benefits that scale with the size of
+data in Kudu tablets. This is a work-in-progress <a href="https://gerrit.cloudera.org/#/c/10983/">patch</a>.
+The implementation in the patch works only for equality predicates on the non-first primary key
+columns. An important point to note is that although, in the above specific example, the number of prefix
+columns is one (<code class="highlighter-rouge">host</code>), this approach is generalized to work with any number of prefix columns.</p>
+
+<p>This work also lays the groundwork to leverage the skip scan approach and optimize query processing time in the
+following use cases:</p>
+
+<ul>
+ <li>Range predicates</li>
+ <li>In-list predicates</li>
+</ul>
+
+<p>This was my first time working on an open source project. I thoroughly enjoyed working on this challenging problem,
+right from understanding the scan path in Kudu to working on a full-fledged implementation of
+the skip scan optimization. I am very grateful to the Kudu team for guiding and supporting me throughout the
+internship period.</p>
+
+<h1 id="references">References</h1>
+
+<p><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/42851.pdf">[1]</a>: Gupta, Ashish, et al. “Mesa:
+Geo-replicated, near real-time, scalable data warehousing.” Proceedings of the VLDB Endowment 7.12 (2014): 1259-1270.</p>
+
+<p><a href="https://oracle-base.com/articles/9i/index-skip-scanning/">[2]</a>: Index Skip Scanning - Oracle Database</p>
+
+<p><a href="https://www.sqlite.org/optoverview.html#skipscan">[3]</a>: Skip Scan - SQLite</p>
+
+
+ </div>
+</article>
+
+
+ </div>
+ <div class="col-lg-3 recent-posts">
+ <h3>Recent posts</h3>
+ <ul>
+
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
+ <li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
+
+ <li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
+
+ <li> <a href="/2018/07/10/instrumentation-in-kudu.html">Instrumentation in Apache Kudu</a> </li>
+
+ <li> <a href="/2018/03/23/apache-kudu-1-7-0-released.html">Apache Kudu 1.7.0 released</a> </li>
+
+ <li> <a href="/2017/12/08/apache-kudu-1-6-0-released.html">Apache Kudu 1.6.0 released</a> </li>
+
+ <li> <a href="/2017/10/23/nosql-kudu-spanner-slides.html">Slides: A brave new world in mutable big data: Relational storage</a> </li>
+
+ <li> <a href="/2017/09/18/kudu-consistency-pt1.html">Consistency in Apache Kudu, Part 1</a> </li>
+
+ <li> <a href="/2017/09/08/apache-kudu-1-5-0-released.html">Apache Kudu 1.5.0 released</a> </li>
+
+ <li> <a href="/2017/06/13/apache-kudu-1-4-0-released.html">Apache Kudu 1.4.0 released</a> </li>
+
+ <li> <a href="/2017/04/19/apache-kudu-1-3-1-released.html">Apache Kudu 1.3.1 released</a> </li>
+
+ <li> <a href="/2017/03/20/apache-kudu-1-3-0-released.html">Apache Kudu 1.3.0 released</a> </li>
+
+ <li> <a href="/2017/01/20/apache-kudu-1-2-0-released.html">Apache Kudu 1.2.0 released</a> </li>
+
+ <li> <a href="/2016/11/15/weekly-update.html">Apache Kudu Weekly Update November 15th, 2016</a> </li>
+
+ <li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
+
+ </ul>
+ </div>
+</div>
+
+ <footer class="footer">
+ <div class="row">
+ <div class="col-md-9">
+ <p class="small">
+ Copyright © 2016 The Apache Software Foundation.
+ </p>
+ <p class="small">
+ Apache Kudu, Kudu, Apache, the Apache feather logo, and the Apache Kudu
+ project logo are either registered trademarks or trademarks of The
+ Apache Software Foundation in the United States and other countries.
+ </p>
+ </div>
+ <div class="col-md-3">
+ <a class="pull-right" href="https://www.apache.org/events/current-event.html">
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ </div>
+ </div>
+ </footer>
+ </div>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+ <script>
+ // Try to detect touch-screen devices. Note: Many laptops have touch screens.
+ $(document).ready(function() {
+ if ("ontouchstart" in document.documentElement) {
+ $(document.documentElement).addClass("touch");
+ } else {
+ $(document.documentElement).addClass("no-touch");
+ }
+ });
+ </script>
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"
+ integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS"
+ crossorigin="anonymous"></script>
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-68448017-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js"></script>
+ <script>
+ anchors.options = {
+ placement: 'right',
+ visible: 'touch',
+ };
+ anchors.add();
+ </script>
+ </body>
+</html>
+
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/index.html
----------------------------------------------------------------------
diff --git a/blog/index.html b/blog/index.html
index cefe6a4..f70bfea 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -117,6 +117,29 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a></h1>
+ <p class="meta">Posted 26 Sep 2018 by Anupama Gupta</p>
+ </header>
+ <div class="entry-content">
+
+ <p>This summer I got the opportunity to intern with the Apache Kudu team at Cloudera.
+My project was to optimize the Kudu scan path by implementing a technique called
+index skip scan (a.k.a. scan-to-seek, see section 4.1 in [1]). I wanted to share
+my experience and the progress we’ve made so far on the approach.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a></h1>
<p class="meta">Posted 11 Sep 2018 by Mac Noland</p>
</header>
@@ -219,31 +242,6 @@ optimizations, incremental improvements, and bug fixes.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2017/12/08/apache-kudu-1-6-0-released.html">Apache Kudu 1.6.0 released</a></h1>
- <p class="meta">Posted 08 Dec 2017 by Mike Percy</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu team is happy to announce the release of Kudu 1.6.0!</p>
-
-<p>Apache Kudu 1.6.0 is a minor release that offers new features, performance
-optimizations, incremental improvements, and bug fixes.</p>
-
-<p>Release highlights:</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2017/12/08/apache-kudu-1-6-0-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -262,6 +260,8 @@ optimizations, incremental improvements, and bug fixes.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -290,8 +290,6 @@ optimizations, incremental improvements, and bug fixes.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/10/index.html
----------------------------------------------------------------------
diff --git a/blog/page/10/index.html b/blog/page/10/index.html
index eff1bfa..6a60e13 100644
--- a/blog/page/10/index.html
+++ b/blog/page/10/index.html
@@ -117,6 +117,29 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Predicate Improvements in Kudu 0.8</a></h1>
+ <p class="meta">Posted 19 Apr 2016 by Dan Burkert</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The recently released Kudu version 0.8 ships with a host of new improvements to
+scan predicates. Performance and usability have been improved, especially for
+tables taking advantage of <a href="http://kudu.apache.org/docs/schema_design.html#data-distribution">advanced partitioning
+options</a>.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/04/18/weekly-update.html">Apache Kudu (incubating) Weekly Update April 18, 2016</a></h1>
<p class="meta">Posted 18 Apr 2016 by Todd Lipcon</p>
</header>
@@ -217,27 +240,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/04/04/weekly-update.html">Apache Kudu (incubating) Weekly Update April 4, 2016</a></h1>
- <p class="meta">Posted 04 Apr 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the third edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/04/04/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -258,6 +260,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -286,8 +290,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/11/index.html
----------------------------------------------------------------------
diff --git a/blog/page/11/index.html b/blog/page/11/index.html
index 259609e..5b12b5e 100644
--- a/blog/page/11/index.html
+++ b/blog/page/11/index.html
@@ -117,6 +117,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/04/04/weekly-update.html">Apache Kudu (incubating) Weekly Update April 4, 2016</a></h1>
+ <p class="meta">Posted 04 Apr 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the third edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu (incubating) project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/04/04/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/03/28/weekly-update.html">Apache Kudu (incubating) Weekly Update March 28, 2016</a></h1>
<p class="meta">Posted 28 Mar 2016 by Todd Lipcon</p>
</header>
@@ -229,6 +250,8 @@ part of the ASF Incubator, version 0.7.0!</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -257,8 +280,6 @@ part of the ASF Incubator, version 0.7.0!</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/2/index.html
----------------------------------------------------------------------
diff --git a/blog/page/2/index.html b/blog/page/2/index.html
index f0c303b..ad6a4af 100644
--- a/blog/page/2/index.html
+++ b/blog/page/2/index.html
@@ -117,6 +117,31 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2017/12/08/apache-kudu-1-6-0-released.html">Apache Kudu 1.6.0 released</a></h1>
+ <p class="meta">Posted 08 Dec 2017 by Mike Percy</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu team is happy to announce the release of Kudu 1.6.0!</p>
+
+<p>Apache Kudu 1.6.0 is a minor release that offers new features, performance
+optimizations, incremental improvements, and bug fixes.</p>
+
+<p>Release highlights:</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2017/12/08/apache-kudu-1-6-0-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2017/10/23/nosql-kudu-spanner-slides.html">Slides: A brave new world in mutable big data: Relational storage</a></h1>
<p class="meta">Posted 23 Oct 2017 by Todd Lipcon</p>
</header>
@@ -217,40 +242,6 @@ improvements, optimizations, and bug fixes.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2017/04/19/apache-kudu-1-3-1-released.html">Apache Kudu 1.3.1 released</a></h1>
- <p class="meta">Posted 19 Apr 2017 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu team is happy to announce the release of Kudu 1.3.1!</p>
-
-<p>Apache Kudu 1.3.1 is a bug fix release which fixes critical issues discovered
-in Apache Kudu 1.3.0. In particular, this fixes a bug in which data could be
-incorrectly deleted after certain sequences of node failures. Several other
-bugs are also fixed. See the release notes for details.</p>
-
-<p>Users of Kudu 1.3.0 are encouraged to upgrade to 1.3.1 immediately.</p>
-
-<ul>
- <li>Download the <a href="/releases/1.3.1/">Kudu 1.3.1 source release</a></li>
- <li>Convenience binary artifacts for the Java client and various Java
-integrations (eg Spark, Flume) are also now available via the ASF Maven
-repository.</li>
-</ul>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2017/04/19/apache-kudu-1-3-1-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -271,6 +262,8 @@ repository.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -299,8 +292,6 @@ repository.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/3/index.html
----------------------------------------------------------------------
diff --git a/blog/page/3/index.html b/blog/page/3/index.html
index c102756..bf91a51 100644
--- a/blog/page/3/index.html
+++ b/blog/page/3/index.html
@@ -117,6 +117,40 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2017/04/19/apache-kudu-1-3-1-released.html">Apache Kudu 1.3.1 released</a></h1>
+ <p class="meta">Posted 19 Apr 2017 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu team is happy to announce the release of Kudu 1.3.1!</p>
+
+<p>Apache Kudu 1.3.1 is a bug fix release which fixes critical issues discovered
+in Apache Kudu 1.3.0. In particular, this fixes a bug in which data could be
+incorrectly deleted after certain sequences of node failures. Several other
+bugs are also fixed. See the release notes for details.</p>
+
+<p>Users of Kudu 1.3.0 are encouraged to upgrade to 1.3.1 immediately.</p>
+
+<ul>
+ <li>Download the <a href="/releases/1.3.1/">Kudu 1.3.1 source release</a></li>
+ <li>Convenience binary artifacts for the Java client and various Java
+integrations (eg Spark, Flume) are also now available via the ASF Maven
+repository.</li>
+</ul>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2017/04/19/apache-kudu-1-3-1-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2017/03/20/apache-kudu-1-3-0-released.html">Apache Kudu 1.3.0 released</a></h1>
<p class="meta">Posted 20 Mar 2017 by Todd Lipcon</p>
</header>
@@ -202,27 +236,6 @@ covers ongoing development and news in the Apache Kudu project.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a></h1>
- <p class="meta">Posted 20 Oct 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the twenty-second edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/10/20/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -243,6 +256,8 @@ covers ongoing development and news in the Apache Kudu project.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -271,8 +286,6 @@ covers ongoing development and news in the Apache Kudu project.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
[2/4] kudu-site git commit: Publish commit(s) from site source repo:
83530755d Blogpost describing index skip scan optimization.
Posted by mp...@apache.org.
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/4/index.html
----------------------------------------------------------------------
diff --git a/blog/page/4/index.html b/blog/page/4/index.html
index 21724ba..3f6dd71 100644
--- a/blog/page/4/index.html
+++ b/blog/page/4/index.html
@@ -117,6 +117,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a></h1>
+ <p class="meta">Posted 20 Oct 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the twenty-second edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/10/20/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/10/11/weekly-update.html">Apache Kudu Weekly Update October 11th, 2016</a></h1>
<p class="meta">Posted 11 Oct 2016 by Todd Lipcon</p>
</header>
@@ -209,320 +230,6 @@ scan path to speed up queries.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a></h1>
- <p class="meta">Posted 31 Aug 2016 by Ara Abrahamian</p>
- </header>
- <div class="entry-content">
-
- <p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
-using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.</p>
-
-<h2 id="why-kudu">Why Kudu</h2>
-
-<p>Traditionally in the Hadoop ecosystem we’ve dealt with various <em>batch processing</em> technologies such
-as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig,
-Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to
-process the whole data set in batches, again and again, as soon as new data gets added. Things get
-really complicated when a few such tasks need to get chained together, or when the same data set
-needs to be processed in various ways by different jobs, while all compete for the shared cluster
-resources.</p>
-
-<p>The opposite of this approach is <em>stream processing</em>: process the data as soon as it arrives, not
-in batches. Streaming systems such as Spark Streaming, Storm, Kafka Streams, and many others make
-this possible. But writing streaming services is not trivial. The streaming systems are becoming
-more and more capable and support more complex constructs, but they are not yet easy to use. All
-queries and processes need to be carefully planned and implemented.</p>
-
-<p>To summarize, <em>batch processing</em> is:</p>
-
-<ul>
- <li>file-based</li>
- <li>a paradigm that processes large chunks of data as a group</li>
- <li>high latency and high throughput, both for ingest and query</li>
- <li>typically easy to program, but hard to orchestrate</li>
- <li>well suited for writing ad-hoc queries, although they are typically high latency</li>
-</ul>
-
-<p>While <em>stream processing</em> is:</p>
-
-<ul>
- <li>a totally different paradigm, which involves single events and time windows instead of large groups of events</li>
- <li>still file-based and not a long-term database</li>
- <li>not batch-oriented, but incremental</li>
- <li>ultra-fast ingest and ultra-fast query (query results basically pre-calculated)</li>
- <li>not so easy to program, relatively easy to orchestrate</li>
- <li>impossible to write ad-hoc queries</li>
-</ul>
-
-<p>And a Kudu-based <em>near real-time</em> approach is:</p>
-
-<ul>
- <li>flexible and expressive, thanks to SQL support via Apache Impala (incubating)</li>
- <li>a table-oriented, mutable data store that feels like a traditional relational database</li>
- <li>very easy to program, you can even pretend it’s good old MySQL</li>
- <li>low-latency and relatively high throughput, both for ingest and query</li>
-</ul>
-
-<p>At Argyle Data, we’re dealing with complex fraud detection scenarios. We need to ingest massive
-amounts of data, run machine learning algorithms and generate reports. When we created our current
-architecture two years ago we decided to opt for a database as the backbone of our system. That
-database is Apache Accumulo. It’s a key-value based database which runs on top of Hadoop HDFS,
-quite similar to HBase but with some important improvements such as cell level security and ease
-of deployment and management. To enable querying of this data for quite complex reporting and
-analytics, we used Presto, a distributed query engine with a pluggable architecture open-sourced
-by Facebook. We wrote a connector for it to let it run queries against the Accumulo database. This
-architecture has served us well, but there were a few problems:</p>
-
-<ul>
- <li>we need to ingest even more massive volumes of data in real-time</li>
- <li>we need to perform complex machine-learning calculations on even larger data-sets</li>
- <li>we need to support ad-hoc queries, plus long-term data warehouse functionality</li>
-</ul>
-
-<p>So, we’ve started gradually moving the core machine-learning pipeline to a streaming based
-solution. This way we can ingest and process larger data-sets faster in the real-time. But then how
-would we take care of ad-hoc queries and long-term persistence? This is where Kudu comes in. While
-the machine learning pipeline ingests and processes real-time data, we store a copy of the same
-ingested data in Kudu for long-term access and ad-hoc queries. Kudu is our <em>data warehouse</em>. By
-using Kudu and Impala, we can retire our in-house Presto connector and rely on Impala’s
-super-fast query engine.</p>
-
-<p>But how would we make sure data is reliably ingested into the streaming pipeline <em>and</em> the
-Kudu-based data warehouse? This is where Apache Flume comes in.</p>
-
-<h2 id="why-flume">Why Flume</h2>
-
-<p>According to their <a href="http://flume.apache.org/">website</a> “Flume is a distributed, reliable, and
-available service for efficiently collecting, aggregating, and moving large amounts of log data.
-It has a simple and flexible architecture based on streaming data flows. It is robust and fault
-tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.” As you
-can see, nowhere is Hadoop mentioned but Flume is typically used for ingesting data to Hadoop
-clusters.</p>
-
-<p><img src="https://blogs.apache.org/flume/mediaresource/ab0d50f6-a960-42cc-971e-3da38ba3adad" alt="png" /></p>
-
-<p>Flume has an extensible architecture. An instance of Flume, called an <em>agent</em>, can have multiple
-<em>channels</em>, with each having multiple <em>sources</em> and <em>sinks</em> of various types. Sources queue data
-in channels, which in turn write out data to sinks. Such <em>pipelines</em> can be chained together to
-create even more complex ones. There may be more than one agent and agents can be configured to
-support failover and recovery.</p>
-
-<p>Flume comes with a bunch of built-in types of channels, sources and sinks. Memory channel is the
-default (an in-memory queue with no persistence to disk), but other options such as Kafka- and
-File-based channels are also provided. As for the sources, Avro, JMS, Thrift, spooling directory
-source are some of the built-in ones. Flume also ships with many sinks, including sinks for writing
-data to HDFS, HBase, Hive, Kafka, as well as to other Flume agents.</p>
-
-<p>In the rest of this post I’ll go over the Kudu Flume sink and show you how to configure Flume to
-write ingested data to a Kudu table. The sink has been part of the Kudu distribution since the 0.8
-release and the source code can be found <a href="https://github.com/apache/kudu/tree/master/java/kudu-flume-sink">here</a>.</p>
-
-<h2 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h2>
-
-<p>Here is a sample flume configuration file:</p>
-
-<pre><code>agent1.sources = source1
-agent1.channels = channel1
-agent1.sinks = sink1
-
-agent1.sources.source1.type = exec
-agent1.sources.source1.command = /usr/bin/vmstat 1
-agent1.sources.source1.channels = channel1
-
-agent1.channels.channel1.type = memory
-agent1.channels.channel1.capacity = 10000
-agent1.channels.channel1.transactionCapacity = 1000
-
-agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
-agent1.sinks.sink1.masterAddresses = localhost
-agent1.sinks.sink1.tableName = stats
-agent1.sinks.sink1.channel = channel1
-agent1.sinks.sink1.batchSize = 50
-agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
-</code></pre>
-
-<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate
-virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel,
-which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using
-<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is
-the built-in and default producer, but it’s implemented as a showcase for how to write Flume
-events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
-need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply
-copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
-<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
-<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
-
-<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
-(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
-Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
-table, it has to be created before the Kudu Flume Sink is started.</p>
-
-<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many
-Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
-impact on ingest performance of the Kudu cluster.</p>
-
-<p>Here is a complete list of KuduSink parameters:</p>
-
-<table>
- <thead>
- <tr>
- <th>Parameter Name</th>
- <th>Default</th>
- <th>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>masterAddresses</td>
- <td>N/A</td>
- <td>Comma-separated list of “host:port” pairs of the masters (port optional)</td>
- </tr>
- <tr>
- <td>tableName</td>
- <td>N/A</td>
- <td>The name of the table in Kudu to write to</td>
- </tr>
- <tr>
- <td>producer</td>
- <td>org.apache.kudu.flume.sink.SimpleKuduEventProducer</td>
- <td>The fully qualified class name of the Kudu event producer the sink should use</td>
- </tr>
- <tr>
- <td>batchSize</td>
- <td>100</td>
- <td>Maximum number of events the sink should take from the channel per transaction, if available</td>
- </tr>
- <tr>
- <td>timeoutMillis</td>
- <td>30000</td>
- <td>Timeout period for Kudu operations, in milliseconds</td>
- </tr>
- <tr>
- <td>ignoreDuplicateRows</td>
- <td>true</td>
- <td>Whether to ignore errors indicating that we attempted to insert duplicate rows into Kudu</td>
- </tr>
- </tbody>
-</table>
-
-<p>Let’s take a look at the source code for the built-in producer class:</p>
-
-<pre><code class="language-java">public class SimpleKuduEventProducer implements KuduEventProducer {
- private byte[] payload;
- private KuduTable table;
- private String payloadColumn;
-
- public SimpleKuduEventProducer(){
- }
-
- @Override
- public void configure(Context context) {
- payloadColumn = context.getString("payloadColumn","payload");
- }
-
- @Override
- public void configure(ComponentConfiguration conf) {
- }
-
- @Override
- public void initialize(Event event, KuduTable table) {
- this.payload = event.getBody();
- this.table = table;
- }
-
- @Override
- public List<Operation> getOperations() throws FlumeException {
- try {
- Insert insert = table.newInsert();
- PartialRow row = insert.getRow();
- row.addBinary(payloadColumn, payload);
-
- return Collections.singletonList((Operation) insert);
- } catch (Exception e){
- throw new FlumeException("Failed to create Kudu Insert object!", e);
- }
- }
-
- @Override
- public void close() {
- }
-}
-</code></pre>
-
-<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
-which itself looks like this:</p>
-
-<pre><code class="language-java">public interface KuduEventProducer extends Configurable, ConfigurableComponent {
- /**
- * Initialize the event producer.
- * @param event to be written to Kudu
- * @param table the KuduTable object used for creating Kudu Operation objects
- */
- void initialize(Event event, KuduTable table);
-
- /**
- * Get the operations that should be written out to Kudu as a result of this
- * event. This list is written to Kudu using the Kudu client API.
- * @return List of {@link org.kududb.client.Operation} which
- * are written as such to Kudu
- */
- List<Operation> getOperations();
-
- /*
- * Clean up any state. This will be called when the sink is being stopped.
- */
- void close();
-}
-</code></pre>
-
-<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated
-by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
-<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
-column which will hold the value of the Flume event payload. If you recall from above, we had
-configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row
-from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table.
-<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
-define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for
-example).</p>
-
-<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In
-SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
-the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used
-<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer
-implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you
-will need to write your own custom producer in the real world, but you can base your implementation
-on the built-in ones.</p>
-
-<p>In the future, we plan to add more flexible event producer implementations so that creation of a
-custom event producer is not required to write data to Kudu. See
-<a href="https://gerrit.cloudera.org/#/c/4034/">here</a> for a work-in-progress generic event producer for
-Avro-encoded Events.</p>
-
-<h2 id="conclusion">Conclusion</h2>
-
-<p>Kudu is a scalable data store which lets us ingest insane amounts of data per second. Apache Flume
-helps us aggregate data from various sources, and the Kudu Flume Sink lets us easily store
-the aggregated Flume events into Kudu. Together they enable us to create a data warehouse out of
-disparate sources.</p>
-
-<p><em>Ara Abrahamian is a software engineer at Argyle Data building fraud detection systems using
-sophisticated machine learning methods. Ara is the original author of the Flume Kudu Sink that
-is included in the Kudu distribution. You can follow him on Twitter at
-<a href="https://twitter.com/ara_e">@ara_e</a>.</em></p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/08/31/intro-flume-kudu-sink.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -543,6 +250,8 @@ is included in the Kudu distribution. You can follow him on Twitter at
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -571,8 +280,6 @@ is included in the Kudu distribution. You can follow him on Twitter at
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/5/index.html
----------------------------------------------------------------------
diff --git a/blog/page/5/index.html b/blog/page/5/index.html
index 1e4c02d..eba0ce9 100644
--- a/blog/page/5/index.html
+++ b/blog/page/5/index.html
@@ -117,6 +117,320 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a></h1>
+ <p class="meta">Posted 31 Aug 2016 by Ara Abrahamian</p>
+ </header>
+ <div class="entry-content">
+
+ <p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered
+using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.</p>
+
+<h2 id="why-kudu">Why Kudu</h2>
+
+<p>Traditionally in the Hadoop ecosystem we’ve dealt with various <em>batch processing</em> technologies such
+as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig,
+Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to
+process the whole data set in batches, again and again, as soon as new data gets added. Things get
+really complicated when a few such tasks need to get chained together, or when the same data set
+needs to be processed in various ways by different jobs, while all compete for the shared cluster
+resources.</p>
+
+<p>The opposite of this approach is <em>stream processing</em>: process the data as soon as it arrives, not
+in batches. Streaming systems such as Spark Streaming, Storm, Kafka Streams, and many others make
+this possible. But writing streaming services is not trivial. The streaming systems are becoming
+more and more capable and support more complex constructs, but they are not yet easy to use. All
+queries and processes need to be carefully planned and implemented.</p>
+
+<p>To summarize, <em>batch processing</em> is:</p>
+
+<ul>
+ <li>file-based</li>
+ <li>a paradigm that processes large chunks of data as a group</li>
+ <li>high latency and high throughput, both for ingest and query</li>
+ <li>typically easy to program, but hard to orchestrate</li>
+ <li>well suited for writing ad-hoc queries, although they are typically high latency</li>
+</ul>
+
+<p>While <em>stream processing</em> is:</p>
+
+<ul>
+ <li>a totally different paradigm, which involves single events and time windows instead of large groups of events</li>
+ <li>still file-based and not a long-term database</li>
+ <li>not batch-oriented, but incremental</li>
+ <li>ultra-fast ingest and ultra-fast query (query results basically pre-calculated)</li>
+ <li>not so easy to program, relatively easy to orchestrate</li>
+ <li>impossible to write ad-hoc queries</li>
+</ul>
+
+<p>And a Kudu-based <em>near real-time</em> approach is:</p>
+
+<ul>
+ <li>flexible and expressive, thanks to SQL support via Apache Impala (incubating)</li>
+ <li>a table-oriented, mutable data store that feels like a traditional relational database</li>
+ <li>very easy to program, you can even pretend it’s good old MySQL</li>
+ <li>low-latency and relatively high throughput, both for ingest and query</li>
+</ul>
+
+<p>At Argyle Data, we’re dealing with complex fraud detection scenarios. We need to ingest massive
+amounts of data, run machine learning algorithms and generate reports. When we created our current
+architecture two years ago we decided to opt for a database as the backbone of our system. That
+database is Apache Accumulo. It’s a key-value based database which runs on top of Hadoop HDFS,
+quite similar to HBase but with some important improvements such as cell level security and ease
+of deployment and management. To enable querying of this data for quite complex reporting and
+analytics, we used Presto, a distributed query engine with a pluggable architecture open-sourced
+by Facebook. We wrote a connector for it to let it run queries against the Accumulo database. This
+architecture has served us well, but there were a few problems:</p>
+
+<ul>
+ <li>we need to ingest even more massive volumes of data in real-time</li>
+ <li>we need to perform complex machine-learning calculations on even larger data-sets</li>
+ <li>we need to support ad-hoc queries, plus long-term data warehouse functionality</li>
+</ul>
+
+<p>So, we’ve started gradually moving the core machine-learning pipeline to a streaming based
+solution. This way we can ingest and process larger data-sets faster in the real-time. But then how
+would we take care of ad-hoc queries and long-term persistence? This is where Kudu comes in. While
+the machine learning pipeline ingests and processes real-time data, we store a copy of the same
+ingested data in Kudu for long-term access and ad-hoc queries. Kudu is our <em>data warehouse</em>. By
+using Kudu and Impala, we can retire our in-house Presto connector and rely on Impala’s
+super-fast query engine.</p>
+
+<p>But how would we make sure data is reliably ingested into the streaming pipeline <em>and</em> the
+Kudu-based data warehouse? This is where Apache Flume comes in.</p>
+
+<h2 id="why-flume">Why Flume</h2>
+
+<p>According to their <a href="http://flume.apache.org/">website</a> “Flume is a distributed, reliable, and
+available service for efficiently collecting, aggregating, and moving large amounts of log data.
+It has a simple and flexible architecture based on streaming data flows. It is robust and fault
+tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.” As you
+can see, nowhere is Hadoop mentioned but Flume is typically used for ingesting data to Hadoop
+clusters.</p>
+
+<p><img src="https://blogs.apache.org/flume/mediaresource/ab0d50f6-a960-42cc-971e-3da38ba3adad" alt="png" /></p>
+
+<p>Flume has an extensible architecture. An instance of Flume, called an <em>agent</em>, can have multiple
+<em>channels</em>, with each having multiple <em>sources</em> and <em>sinks</em> of various types. Sources queue data
+in channels, which in turn write out data to sinks. Such <em>pipelines</em> can be chained together to
+create even more complex ones. There may be more than one agent and agents can be configured to
+support failover and recovery.</p>
+
+<p>Flume comes with a bunch of built-in types of channels, sources and sinks. Memory channel is the
+default (an in-memory queue with no persistence to disk), but other options such as Kafka- and
+File-based channels are also provided. As for the sources, Avro, JMS, Thrift, spooling directory
+source are some of the built-in ones. Flume also ships with many sinks, including sinks for writing
+data to HDFS, HBase, Hive, Kafka, as well as to other Flume agents.</p>
+
+<p>In the rest of this post I’ll go over the Kudu Flume sink and show you how to configure Flume to
+write ingested data to a Kudu table. The sink has been part of the Kudu distribution since the 0.8
+release and the source code can be found <a href="https://github.com/apache/kudu/tree/master/java/kudu-flume-sink">here</a>.</p>
+
+<h2 id="configuring-the-kudu-flume-sink">Configuring the Kudu Flume Sink</h2>
+
+<p>Here is a sample flume configuration file:</p>
+
+<div class="highlighter-rouge">agent1.sources = source1
+agent1.channels = channel1
+agent1.sinks = sink1
+
+agent1.sources.source1.type = exec
+agent1.sources.source1.command = /usr/bin/vmstat 1
+agent1.sources.source1.channels = channel1
+
+agent1.channels.channel1.type = memory
+agent1.channels.channel1.capacity = 10000
+agent1.channels.channel1.transactionCapacity = 1000
+
+agent1.sinks.sink1.type = org.apache.flume.sink.kudu.KuduSink
+agent1.sinks.sink1.masterAddresses = localhost
+agent1.sinks.sink1.tableName = stats
+agent1.sinks.sink1.channel = channel1
+agent1.sinks.sink1.batchSize = 50
+agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer
+</div>
+
+<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate
+virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel,
+which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using
+<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is
+the built-in and default producer, but it’s implemented as a showcase for how to write Flume
+events into Kudu tables. For any serious functionality we’d have to write a custom producer. We
+need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply
+copying the <code class="highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the
+<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains
+<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p>
+
+<p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are
+(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing
+Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this
+table, it has to be created before the Kudu Flume Sink is started.</p>
+
+<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many
+Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge
+impact on ingest performance of the Kudu cluster.</p>
+
+<p>Here is a complete list of KuduSink parameters:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Parameter Name</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>masterAddresses</td>
+ <td>N/A</td>
+ <td>Comma-separated list of “host:port” pairs of the masters (port optional)</td>
+ </tr>
+ <tr>
+ <td>tableName</td>
+ <td>N/A</td>
+ <td>The name of the table in Kudu to write to</td>
+ </tr>
+ <tr>
+ <td>producer</td>
+ <td>org.apache.kudu.flume.sink.SimpleKuduEventProducer</td>
+ <td>The fully qualified class name of the Kudu event producer the sink should use</td>
+ </tr>
+ <tr>
+ <td>batchSize</td>
+ <td>100</td>
+ <td>Maximum number of events the sink should take from the channel per transaction, if available</td>
+ </tr>
+ <tr>
+ <td>timeoutMillis</td>
+ <td>30000</td>
+ <td>Timeout period for Kudu operations, in milliseconds</td>
+ </tr>
+ <tr>
+ <td>ignoreDuplicateRows</td>
+ <td>true</td>
+ <td>Whether to ignore errors indicating that we attempted to insert duplicate rows into Kudu</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>Let’s take a look at the source code for the built-in producer class:</p>
+
+<div class="highlighter-rouge"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span>
+
+ <span class="kd">public</span> <span class="nf">SimpleKuduEventProducer</span><span class="o">(){</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span>
+ <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span>
+ <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span>
+
+ <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span>
+ <span class="k">throw</span> <span class="k">new</span> <span class="nf">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">close</span><span class="o">()</span> <span class="o">{</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</div>
+
+<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface,
+which itself looks like this:</p>
+
+<div class="highlighter-rouge"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span>
+ <span class="cm">/**
+ * Initialize the event producer.
+ * @param event to be written to Kudu
+ * @param table the KuduTable object used for creating Kudu Operation objects
+ */</span>
+ <span class="kt">void</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span>
+
+ <span class="cm">/**
+ * Get the operations that should be written out to Kudu as a result of this
+ * event. This list is written to Kudu using the Kudu client API.
+ * @return List of {@link org.kududb.client.Operation} which
+ * are written as such to Kudu
+ */</span>
+ <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="nf">getOperations</span><span class="o">();</span>
+
+ <span class="cm">/*
+ * Clean up any state. This will be called when the sink is being stopped.
+ */</span>
+ <span class="kt">void</span> <span class="nf">close</span><span class="o">();</span>
+<span class="o">}</span>
+</div>
+
+<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated
+by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named
+<code class="highlighter-rouge">payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the
+column which will hold the value of the Flume event payload. If you recall from above, we had
+configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row
+from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table.
+<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would
+define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for
+example).</p>
+
+<p>The main producer logic resides in the <code class="highlighter-rouge">public List<Operation> getOperations()</code> method. In
+SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into
+the Kudu table. Here we call Kudu’s <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used
+<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact there’s another producer
+implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you
+will need to write your own custom producer in the real world, but you can base your implementation
+on the built-in ones.</p>
+
+<p>In the future, we plan to add more flexible event producer implementations so that creation of a
+custom event producer is not required to write data to Kudu. See
+<a href="https://gerrit.cloudera.org/#/c/4034/">here</a> for a work-in-progress generic event producer for
+Avro-encoded Events.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>Kudu is a scalable data store which lets us ingest insane amounts of data per second. Apache Flume
+helps us aggregate data from various sources, and the Kudu Flume Sink lets us easily store
+the aggregated Flume events into Kudu. Together they enable us to create a data warehouse out of
+disparate sources.</p>
+
+<p><em>Ara Abrahamian is a software engineer at Argyle Data building fraud detection systems using
+sophisticated machine learning methods. Ara is the original author of the Flume Kudu Sink that
+is included in the Kudu distribution. You can follow him on Twitter at
+<a href="https://twitter.com/ara_e">@ara_e</a>.</em></p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/08/31/intro-flume-kudu-sink.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a></h1>
<p class="meta">Posted 23 Aug 2016 by Dan Burkert</p>
</header>
@@ -201,27 +515,6 @@ covers ongoing development and news in the Apache Kudu project.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/07/26/weekly-update.html">Apache Kudu Weekly Update July 26, 2016</a></h1>
- <p class="meta">Posted 26 Jul 2016 by Jean-Daniel Cryans</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the eighteenth edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/07/26/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -242,6 +535,8 @@ covers ongoing development and news in the Apache Kudu project.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -270,8 +565,6 @@ covers ongoing development and news in the Apache Kudu project.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/6/index.html
----------------------------------------------------------------------
diff --git a/blog/page/6/index.html b/blog/page/6/index.html
index 5801003..b2b5e52 100644
--- a/blog/page/6/index.html
+++ b/blog/page/6/index.html
@@ -117,6 +117,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/07/26/weekly-update.html">Apache Kudu Weekly Update July 26, 2016</a></h1>
+ <p class="meta">Posted 26 Jul 2016 by Jean-Daniel Cryans</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the eighteenth edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/07/26/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/07/25/asf-graduation.html">The Apache Software Foundation Announces Apache® Kudu™ as a Top-Level Project</a></h1>
<p class="meta">Posted 25 Jul 2016 by Jean-Daniel Cryans</p>
</header>
@@ -209,27 +230,6 @@ of 0.9.0 are encouraged to update to the new version at their earliest convenien
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/06/27/weekly-update.html">Apache Kudu (incubating) Weekly Update June 27, 2016</a></h1>
- <p class="meta">Posted 27 Jun 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the fifteenth edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/06/27/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -250,6 +250,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -278,8 +280,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/7/index.html
----------------------------------------------------------------------
diff --git a/blog/page/7/index.html b/blog/page/7/index.html
index d0dfe49..0692c27 100644
--- a/blog/page/7/index.html
+++ b/blog/page/7/index.html
@@ -117,6 +117,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/06/27/weekly-update.html">Apache Kudu (incubating) Weekly Update June 27, 2016</a></h1>
+ <p class="meta">Posted 27 Jun 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the fifteenth edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu (incubating) project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/06/27/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/06/24/multi-master-1-0-0.html">Master fault tolerance in Kudu 1.0</a></h1>
<p class="meta">Posted 24 Jun 2016 by Adar Dembo</p>
</header>
@@ -202,37 +223,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/06/10/apache-kudu-0-9-0-released.html">Apache Kudu (incubating) 0.9.0 released</a></h1>
- <p class="meta">Posted 10 Jun 2016 by Jean-Daniel Cryans</p>
- </header>
- <div class="entry-content">
-
- <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu
-0.9.0!</p>
-
-<p>This latest version adds basic UPSERT functionality and an improved Apache Spark Data Source
-that doesn’t rely on the MapReduce I/O formats. It also improves Tablet Server
-restart time as well as write performance under high load. Finally, Kudu now enforces
-the specification of a partitioning scheme for new tables.</p>
-
-<ul>
- <li>Read the detailed <a href="http://kudu.apache.org/releases/0.9.0/docs/release_notes.html">Kudu 0.9.0 release notes</a></li>
- <li>Download the <a href="http://kudu.apache.org/releases/0.9.0/">Kudu 0.9.0 source release</a></li>
-</ul>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/06/10/apache-kudu-0-9-0-released.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -253,6 +243,8 @@ the specification of a partitioning scheme for new tables.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -281,8 +273,6 @@ the specification of a partitioning scheme for new tables.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/8/index.html
----------------------------------------------------------------------
diff --git a/blog/page/8/index.html b/blog/page/8/index.html
index ce0a7e1..aa53f05 100644
--- a/blog/page/8/index.html
+++ b/blog/page/8/index.html
@@ -117,6 +117,37 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/06/10/apache-kudu-0-9-0-released.html">Apache Kudu (incubating) 0.9.0 released</a></h1>
+ <p class="meta">Posted 10 Jun 2016 by Jean-Daniel Cryans</p>
+ </header>
+ <div class="entry-content">
+
+ <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu
+0.9.0!</p>
+
+<p>This latest version adds basic UPSERT functionality and an improved Apache Spark Data Source
+that doesn’t rely on the MapReduce I/O formats. It also improves Tablet Server
+restart time as well as write performance under high load. Finally, Kudu now enforces
+the specification of a partitioning scheme for new tables.</p>
+
+<ul>
+ <li>Read the detailed <a href="http://kudu.apache.org/releases/0.9.0/docs/release_notes.html">Kudu 0.9.0 release notes</a></li>
+ <li>Download the <a href="http://kudu.apache.org/releases/0.9.0/">Kudu 0.9.0 source release</a></li>
+</ul>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/06/10/apache-kudu-0-9-0-released.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/06/06/weekly-update.html">Apache Kudu (incubating) Weekly Update June 6, 2016</a></h1>
<p class="meta">Posted 06 Jun 2016 by Jean-Daniel Cryans</p>
</header>
@@ -200,27 +231,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/05/16/weekly-update.html">Apache Kudu (incubating) Weekly Update May 16, 2016</a></h1>
- <p class="meta">Posted 16 May 2016 by Todd Lipcon</p>
- </header>
- <div class="entry-content">
-
- <p>Welcome to the ninth edition of the Kudu Weekly Update. This weekly blog post
-covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/05/16/weekly-update.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -241,6 +251,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -269,8 +281,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/blog/page/9/index.html
----------------------------------------------------------------------
diff --git a/blog/page/9/index.html b/blog/page/9/index.html
index ce14d37..85c617d 100644
--- a/blog/page/9/index.html
+++ b/blog/page/9/index.html
@@ -117,6 +117,27 @@
<!-- Articles -->
<article>
<header>
+ <h1 class="entry-title"><a href="/2016/05/16/weekly-update.html">Apache Kudu (incubating) Weekly Update May 16, 2016</a></h1>
+ <p class="meta">Posted 16 May 2016 by Todd Lipcon</p>
+ </header>
+ <div class="entry-content">
+
+ <p>Welcome to the ninth edition of the Kudu Weekly Update. This weekly blog post
+covers ongoing development and news in the Apache Kudu (incubating) project.</p>
+
+
+
+ </div>
+ <div class="read-full">
+ <a class="btn btn-info" href="/2016/05/16/weekly-update.html">Read full post...</a>
+ </div>
+</article>
+
+
+
+<!-- Articles -->
+<article>
+ <header>
<h1 class="entry-title"><a href="/2016/05/09/weekly-update.html">Apache Kudu (incubating) Weekly Update May 9, 2016</a></h1>
<p class="meta">Posted 09 May 2016 by Jean-Daniel Cryans</p>
</header>
@@ -197,29 +218,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p>
-<!-- Articles -->
-<article>
- <header>
- <h1 class="entry-title"><a href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Predicate Improvements in Kudu 0.8</a></h1>
- <p class="meta">Posted 19 Apr 2016 by Dan Burkert</p>
- </header>
- <div class="entry-content">
-
- <p>The recently released Kudu version 0.8 ships with a host of new improvements to
-scan predicates. Performance and usability have been improved, especially for
-tables taking advantage of <a href="http://kudu.apache.org/docs/schema_design.html#data-distribution">advanced partitioning
-options</a>.</p>
-
-
-
- </div>
- <div class="read-full">
- <a class="btn btn-info" href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Read full post...</a>
- </div>
-</article>
-
-
-
<!-- Pagination links -->
<nav>
@@ -240,6 +238,8 @@ options</a>.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -268,8 +268,6 @@ options</a>.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/faq.html
----------------------------------------------------------------------
diff --git a/faq.html b/faq.html
index 5cd722f..65885fa 100644
--- a/faq.html
+++ b/faq.html
@@ -345,8 +345,8 @@ enforcing “external consistency” in two different ways: one that optimizes f
requires the user to perform additional work and another that requires no additional
work but can result in some additional latency.</li>
<li>Scans have “Read Committed” consistency by default. If the user requires strict-serializable
-scans it can choose the <code>READ_AT_SNAPSHOT</code> mode and, optionally, provide a timestamp. The default
-option is non-blocking but the <code>READ_AT_SNAPSHOT</code> option may block when reading from non-leader
+scans it can choose the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> mode and, optionally, provide a timestamp. The default
+option is non-blocking but the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> option may block when reading from non-leader
replicas.</li>
</ul>
@@ -369,7 +369,7 @@ further information and caveats.</p>
<p>Kudu provides direct access via Java and C++ APIs. An experimental Python API is
also available and is expected to be fully supported in the future. The easiest
-way to load data into Kudu is to use a <code>CREATE TABLE ... AS SELECT * FROM ...</code>
+way to load data into Kudu is to use a <code class="highlighter-rouge">CREATE TABLE ... AS SELECT * FROM ...</code>
statement in Impala. Although Kudu has not been extensively tested to work with
ingest tools such as Flume, Sqoop, or Kafka, several of these have been
experimentally tested. Explicit support for these ingest tools is expected with
@@ -378,7 +378,7 @@ Kudu’s first generally available release.</p>
<h4 id="whats-the-most-efficient-way-to-bulk-load-data-into-kudu">What’s the most efficient way to bulk load data into Kudu?</h4>
<p>The easiest way to load data into Kudu is if the data is already managed by Impala.
-In this case, a simple <code>INSERT INTO TABLE some_kudu_table SELECT * FROM some_csv_table</code>
+In this case, a simple <code class="highlighter-rouge">INSERT INTO TABLE some_kudu_table SELECT * FROM some_csv_table</code>
does the trick.</p>
<p>You can also use Kudu’s MapReduce OutputFormat to load data from HDFS, HBase, or
@@ -530,8 +530,8 @@ features.</p>
Impala can help if you have it available. You can use it to copy your data into
Parquet format using a statement like:</p>
-<pre><code>INSERT INTO TABLE some_parquet_table SELECT * FROM kudu_table
-</code></pre>
+<div class="highlighter-rouge">INSERT INTO TABLE some_parquet_table SELECT * FROM kudu_table
+</div>
<p>then use <a href="http://hadoop.apache.org/docs/r1.2.1/distcp2.html">distcp</a>
to copy the Parquet data to another cluster.</p>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index 218dfab..49afcef 100644
--- a/feed.xml
+++ b/feed.xml
@@ -1,4 +1,107 @@
-<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2018-09-11T17:54:59+02:00</updated><id>/</id><entry><title>Simplified Data Pipelines with Kudu</title><link href="/2018/09/11/simplified-pipelines-with-kudu.html" rel="alternate" type="text/html" title="Simplified Data Pipelines with Kudu" /><published>2018-09-11T00:00:00+02:00</published><updated>2018-09-11T00:00:00+02:00</updated><id>/2018/09/11/simplified-pipelines-with-kudu</id><content type="html" xml:base="/2018/09/11/simplified-pipelines-with-kudu.html"><p>I’ve been working with Hadoop now for over seven years and fortunately, or unfortunately, have run
+<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="http://jekyllrb.com" version="2.5.3">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2018-09-26T10:55:43-07:00</updated><id>/</id><entry><title>Index Skip Scan Optimization in Kudu</title><link href="/2018/09/26/index-skip-scan-optimization-in-kudu.html" rel="alternate" type="text/html" title="Index Skip Scan Optimization in Kudu" /><published>2018-09-26T00:00:00-07:00</published><updated>2018-09-26T00:00:00-07:00</updated><id>/2018/09/26/index-skip-scan-optimization-in-kudu</id><content type="html" xml:base="/2018/09/26/index-skip-scan-optimization-in-kudu.html"><p>This summer I got the opportunity to intern with the Apache Kudu team at Cloudera.
+My project was to optimize the Kudu scan path by implementing a technique called
+index skip scan (a.k.a. scan-to-seek, see section 4.1 in [1]). I wanted to share
+my experience and the progress we’ve made so far on the approach.</p>
+
+<!--more-->
+
+<p>Let’s begin with discussing the current query flow in Kudu.
+Consider the following table:</p>
+
+<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">metrics</span> <span class="p">(</span>
+ <span class="k">host</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="n">tstamp</span> <span class="nb">INT</span><span class="p">,</span>
+ <span class="n">clusterid</span> <span class="nb">INT</span><span class="p">,</span>
+ <span class="k">role</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">(</span><span class="k">host</span><span class="p">,</span> <span class="n">tstamp</span><span class="p">,</span> <span class="n">clusterid</span><span class="p">)</span>
+<span class="p">);</span></code></pre></div>
+
+<p><img src="/img/index-skip-scan/example-table.png" alt="png" class="img-responsive" />
+<em>Sample rows of table <code class="highlighter-rouge">metrics</code> (sorted by key columns).</em></p>
+
+<p>In this case, by default, Kudu internally builds a primary key index (implemented as a
+<a href="https://en.wikipedia.org/wiki/B-tree">B-tree</a>) for the table <code class="highlighter-rouge">metrics</code>.
+As shown in the table above, the index data is sorted by the composite of all key columns.
+When the user query contains the first key column (<code class="highlighter-rouge">host</code>), Kudu uses the index (as the index data is
+primarily sorted on the first key column).</p>
+
+<p>Now, what if the user query does not contain the first key column and instead only contains the <code class="highlighter-rouge">tstamp</code> column?
+In the above case, the <code class="highlighter-rouge">tstamp</code> column values are sorted with respect to <code class="highlighter-rouge">host</code>,
+but are not globally sorted, and as such, it’s non-trivial to use the index to filter rows.
+Instead, a full tablet scan is done by default. Other databases may optimize such scans by building secondary indexes
+(though it might be redundant to build one on one of the primary keys). However, this isn’t an option for Kudu,
+given its lack of secondary index support.</p>
+
+<p>The question is, can Kudu do better than a full tablet scan here?</p>
+
+<p>The answer is yes! Let’s observe the column preceding the <code class="highlighter-rouge">tstamp</code> column. We will refer to it as the
+“prefix column” and its specific value as the “prefix key”. In this example, <code class="highlighter-rouge">host</code> is the prefix column.
+Note that the prefix keys are sorted in the index and that all rows of a given prefix key are also sorted by the
+remaining key columns. Therefore, we can use the index to skip to the rows that have distinct prefix keys,
+and also satisfy the predicate on the <code class="highlighter-rouge">tstamp</code> column.
+For example, consider the query:</p>
+
+<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="n">clusterid</span> <span class="k">FROM</span> <span class="n">metrics</span> <span class="k">WHERE</span> <span class="n">tstamp</span> <span class="o">=</span> <span class="mi">100</span><span class="p">;</span></code></pre></div>
+
+<p><img src="/img/index-skip-scan/skip-scan-example-table.png" alt="png" class="img-responsive" />
+<em>Skip scan flow illustration. The rows in green are scanned and the rest are skipped.</em></p>
+
+<p>The tablet server can use the index to <strong>skip</strong> to the first row with a distinct prefix key (<code class="highlighter-rouge">host = helium</code>) that
+matches the predicate (<code class="highlighter-rouge">tstamp = 100</code>) and then <strong>scan</strong> through the rows until the predicate no longer matches. At that
+point we would know that no more rows with <code class="highlighter-rouge">host = helium</code> will satisfy the predicate, and we can skip to the next
+prefix key. This holds true for all distinct keys of <code class="highlighter-rouge">host</code>. Hence, this method is popularly known as
+<strong>skip scan optimization</strong>[2, 3].</p>
+
+<h1 id="performance">Performance</h1>
+
+<p>This optimization can speed up queries significantly, depending on the cardinality (number of distinct values) of the
+prefix column. The lower the prefix column cardinality, the better the skip scan performance. In fact, when the
+prefix column cardinality is high, skip scan is not a viable approach. The performance graph (obtained using the example
+schema and query pattern mentioned earlier) is shown below.</p>
+
+<p>Based on our experiments, on up to 10 million rows per tablet (as shown below), we found that the skip scan performance
+begins to get worse with respect to the full tablet scan performance when the prefix column cardinality
+exceeds sqrt(number_of_rows_in_tablet).
+Therefore, in order to use skip scan performance benefits when possible and maintain a consistent performance in cases
+of large prefix column cardinality, we have tentatively chosen to dynamically disable skip scan when the number of skips for
+distinct prefix keys exceeds sqrt(number_of_rows_in_tablet).
+It will be an interesting project to further explore sophisticated heuristics to decide when
+to dynamically disable skip scan.</p>
+
+<p><img src="/img/index-skip-scan/skip-scan-performance-graph.png" alt="png" class="img-responsive" /></p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>Skip scan optimization in Kudu can lead to huge performance benefits that scale with the size of
+data in Kudu tablets. This is a work-in-progress <a href="https://gerrit.cloudera.org/#/c/10983/">patch</a>.
+The implementation in the patch works only for equality predicates on the non-first primary key
+columns. An important point to note is that although, in the above specific example, the number of prefix
+columns is one (<code class="highlighter-rouge">host</code>), this approach is generalized to work with any number of prefix columns.</p>
+
+<p>This work also lays the groundwork to leverage the skip scan approach and optimize query processing time in the
+following use cases:</p>
+
+<ul>
+ <li>Range predicates</li>
+ <li>In-list predicates</li>
+</ul>
+
+<p>This was my first time working on an open source project. I thoroughly enjoyed working on this challenging problem,
+right from understanding the scan path in Kudu to working on a full-fledged implementation of
+the skip scan optimization. I am very grateful to the Kudu team for guiding and supporting me throughout the
+internship period.</p>
+
+<h1 id="references">References</h1>
+
+<p><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/42851.pdf">[1]</a>: Gupta, Ashish, et al. “Mesa:
+Geo-replicated, near real-time, scalable data warehousing.” Proceedings of the VLDB Endowment 7.12 (2014): 1259-1270.</p>
+
+<p><a href="https://oracle-base.com/articles/9i/index-skip-scanning/">[2]</a>: Index Skip Scanning - Oracle Database</p>
+
+<p><a href="https://www.sqlite.org/optoverview.html#skipscan">[3]</a>: Skip Scan - SQLite</p></content><author><name>Anupama Gupta</name></author><summary>This summer I got the opportunity to intern with the Apache Kudu team at Cloudera.
+My project was to optimize the Kudu scan path by implementing a technique called
+index skip scan (a.k.a. scan-to-seek, see section 4.1 in [1]). I wanted to share
+my experience and the progress we’ve made so far on the approach.</summary></entry><entry><title>Simplified Data Pipelines with Kudu</title><link href="/2018/09/11/simplified-pipelines-with-kudu.html" rel="alternate" type="text/html" title="Simplified Data Pipelines with Kudu" /><published>2018-09-11T00:00:00-07:00</published><updated>2018-09-11T00:00:00-07:00</updated><id>/2018/09/11/simplified-pipelines-with-kudu</id><content type="html" xml:base="/2018/09/11/simplified-pipelines-with-kudu.html"><p>I’ve been working with Hadoop now for over seven years and fortunately, or unfortunately, have run
across a lot of structured data use cases. What we, at <a href="https://phdata.io/">phData</a>, have found is
that end users are typically comfortable with tabular data and prefer to access their data in a
structured manner using tables.
@@ -38,7 +141,7 @@ and users to focus on solving business problems, rather than being bothered by t
the backend.</p></content><author><name>Mac Noland</name></author><summary>I’ve been working with Hadoop now for over seven years and fortunately, or unfortunately, have run
across a lot of structured data use cases. What we, at phData, have found is
that end users are typically comfortable with tabular data and prefer to access their data in a
-structured manner using tables.</summary></entry><entry><title>Getting Started with Kudu - an O’Reilly Title</title><link href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html" rel="alternate" type="text/html" title="Getting Started with Kudu - an O'Reilly Title" /><published>2018-08-06T00:00:00+02:00</published><updated>2018-08-06T00:00:00+02:00</updated><id>/2018/08/06/getting-started-with-kudu-an-oreilly-title</id><content type="html" xml:base="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html"><p>The following article by Brock Noland was reposted from the
+structured manner using tables.</summary></entry><entry><title>Getting Started with Kudu - an O’Reilly Title</title><link href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html" rel="alternate" type="text/html" title="Getting Started with Kudu - an O'Reilly Title" /><published>2018-08-06T00:00:00-07:00</published><updated>2018-08-06T00:00:00-07:00</updated><id>/2018/08/06/getting-started-with-kudu-an-oreilly-title</id><content type="html" xml:base="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html"><p>The following article by Brock Noland was reposted from the
<a href="https://www.phdata.io/getting-started-with-kudu/">phData</a>
blog with their permission.</p>
@@ -52,9 +155,9 @@ challenge at that time.
In that context, on October 11th 2012 Todd Lipcon perform Apache Kudu’s initial
commit. The commit message was:</p>
-<pre><code>Code for writing cfiles seems to basically work
+<div class="highlighter-rouge">Code for writing cfiles seems to basically work
Need to write code for reading cfiles, still
-</code></pre>
+</div>
<p>And Kudu development was off and running. Around this same time Todd, on his
internal Wiki page, started listing out the papers he was reading to develop
@@ -90,7 +193,7 @@ of Kudu. Specifically you will learn:</p>
<p>Looking forward, I am excited to see Kudu gain additional features and adoption
and eventually the second revision of this title. In the meantime, if you have
-feedback or questions, please reach out on the <code>#getting-started-kudu</code> channel of
+feedback or questions, please reach out on the <code class="highlighter-rouge">#getting-started-kudu</code> channel of
the <a href="https://getkudu-slack.herokuapp.com/">Kudu Slack</a> or if you prefer non-real-time
communication, please use the user@ mailing list!</p></content><author><name>Brock Noland</name></author><summary>The following article by Brock Noland was reposted from the
phData
@@ -101,7 +204,7 @@ Hadoop platform was hard. Organizations required strong Software Engineering
capabilities to successfully implement complex Lambda architectures or even
simply implement continuous ingest. Updating or deleting data, were simply a
nightmare. General Data Protection Regulation (GDPR) would have been an extreme
-challenge at that time.</summary></entry><entry><title>Instrumentation in Apache Kudu</title><link href="/2018/07/10/instrumentation-in-kudu.html" rel="alternate" type="text/html" title="Instrumentation in Apache Kudu" /><published>2018-07-10T00:00:00+02:00</published><updated>2018-07-10T00:00:00+02:00</updated><id>/2018/07/10/instrumentation-in-kudu</id><content type="html" xml:base="/2018/07/10/instrumentation-in-kudu.html"><p>Last week, the <a href="http://opentracing.io/">OpenTracing</a> community invited me to
+challenge at that time.</summary></entry><entry><title>Instrumentation in Apache Kudu</title><link href="/2018/07/10/instrumentation-in-kudu.html" rel="alternate" type="text/html" title="Instrumentation in Apache Kudu" /><published>2018-07-10T00:00:00-07:00</published><updated>2018-07-10T00:00:00-07:00</updated><id>/2018/07/10/instrumentation-in-kudu</id><content type="html" xml:base="/2018/07/10/instrumentation-in-kudu.html"><p>Last week, the <a href="http://opentracing.io/">OpenTracing</a> community invited me to
their monthly Google Hangout meetup to give an informal talk on tracing and
instrumentation in Apache Kudu.</p>
@@ -136,7 +239,7 @@ While Kudu doesn’t currently support distributed tracing using OpenTracing,
it does have quite a lot of other types of instrumentation, metrics, and
diagnostics logging. The OpenTracing team was interested to hear about some of
the approaches that Kudu has used, and so I gave a brief introduction to topics
-including:</summary></entry><entry><title>Apache Kudu 1.7.0 released</title><link href="/2018/03/23/apache-kudu-1-7-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.7.0 released" /><published>2018-03-23T00:00:00+01:00</published><updated>2018-03-23T00:00:00+01:00</updated><id>/2018/03/23/apache-kudu-1-7-0-released</id><content type="html" xml:base="/2018/03/23/apache-kudu-1-7-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.7.0!</p>
+including:</summary></entry><entry><title>Apache Kudu 1.7.0 released</title><link href="/2018/03/23/apache-kudu-1-7-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.7.0 released" /><published>2018-03-23T00:00:00-07:00</published><updated>2018-03-23T00:00:00-07:00</updated><id>/2018/03/23/apache-kudu-1-7-0-released</id><content type="html" xml:base="/2018/03/23/apache-kudu-1-7-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.7.0!</p>
<p>Apache Kudu 1.7.0 is a minor release that offers new features, performance
optimizations, incremental improvements, and bug fixes.</p>
@@ -207,7 +310,7 @@ Maven repository and are
Apache Kudu 1.7.0 is a minor release that offers new features, performance
optimizations, incremental improvements, and bug fixes.
-Release highlights:</summary></entry><entry><title>Apache Kudu 1.6.0 released</title><link href="/2017/12/08/apache-kudu-1-6-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.6.0 released" /><published>2017-12-08T00:00:00+01:00</published><updated>2017-12-08T00:00:00+01:00</updated><id>/2017/12/08/apache-kudu-1-6-0-released</id><content type="html" xml:base="/2017/12/08/apache-kudu-1-6-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.6.0!</p>
+Release highlights:</summary></entry><entry><title>Apache Kudu 1.6.0 released</title><link href="/2017/12/08/apache-kudu-1-6-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.6.0 released" /><published>2017-12-08T00:00:00-08:00</published><updated>2017-12-08T00:00:00-08:00</updated><id>/2017/12/08/apache-kudu-1-6-0-released</id><content type="html" xml:base="/2017/12/08/apache-kudu-1-6-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.6.0!</p>
<p>Apache Kudu 1.6.0 is a minor release that offers new features, performance
optimizations, incremental improvements, and bug fixes.</p>
@@ -266,7 +369,7 @@ Maven repository and are
Apache Kudu 1.6.0 is a minor release that offers new features, performance
optimizations, incremental improvements, and bug fixes.
-Release highlights:</summary></entry><entry><title>Slides: A brave new world in mutable big data: Relational storage</title><link href="/2017/10/23/nosql-kudu-spanner-slides.html" rel="alternate" type="text/html" title="Slides: A brave new world in mutable big data: Relational storage" /><published>2017-10-23T00:00:00+02:00</published><updated>2017-10-23T00:00:00+02:00</updated><id>/2017/10/23/nosql-kudu-spanner-slides</id><content type="html" xml:base="/2017/10/23/nosql-kudu-spanner-slides.html"><p>Since the Apache Kudu project made its debut in 2015, there have been
+Release highlights:</summary></entry><entry><title>Slides: A brave new world in mutable big data: Relational storage</title><link href="/2017/10/23/nosql-kudu-spanner-slides.html" rel="alternate" type="text/html" title="Slides: A brave new world in mutable big data: Relational storage" /><published>2017-10-23T00:00:00-07:00</published><updated>2017-10-23T00:00:00-07:00</updated><id>/2017/10/23/nosql-kudu-spanner-slides</id><content type="html" xml:base="/2017/10/23/nosql-kudu-spanner-slides.html"><p>Since the Apache Kudu project made its debut in 2015, there have been
a few common questions that kept coming up at every presentation:</p>
<ul>
@@ -326,7 +429,7 @@ a few common questions that kept coming up at every presentation:
Is Kudu an open source version of Google’s Spanner system?
Is Kudu NoSQL or SQL?
- Why does Kudu have a relational data model? Isn’t SQL dead?</summary></entry><entry><title>Consistency in Apache Kudu, Part 1</title><link href="/2017/09/18/kudu-consistency-pt1.html" rel="alternate" type="text/html" title="Consistency in Apache Kudu, Part 1" /><published>2017-09-18T00:00:00+02:00</published><updated>2017-09-18T00:00:00+02:00</updated><id>/2017/09/18/kudu-consistency-pt1</id><content type="html" xml:base="/2017/09/18/kudu-consistency-pt1.html"><p>In this series of short blog posts we will introduce Kudu’s consistency model,
+ Why does Kudu have a relational data model? Isn’t SQL dead?</summary></entry><entry><title>Consistency in Apache Kudu, Part 1</title><link href="/2017/09/18/kudu-consistency-pt1.html" rel="alternate" type="text/html" title="Consistency in Apache Kudu, Part 1" /><published>2017-09-18T00:00:00-07:00</published><updated>2017-09-18T00:00:00-07:00</updated><id>/2017/09/18/kudu-consistency-pt1</id><content type="html" xml:base="/2017/09/18/kudu-consistency-pt1.html"><p>In this series of short blog posts we will introduce Kudu’s consistency model,
its design and ultimate goals, current features, and next steps.
On the way, we’ll shed some light on the more relevant components and how they
fit together.</p>
@@ -445,29 +548,29 @@ have increasing timestamps, depending on the user’s choices.</p>
<p>Row mutations performed by a single client <em>instance</em> are guaranteed to have increasing timestamps
thus reflecting their potential causal relationship. This property is always enforced. However
there are two major <em>“knobs”</em> that are available to the user to make performance trade-offs, the
-<code>Read</code> mode, and the <code>External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a>
+<code class="highlighter-rouge">Read</code> mode, and the <code class="highlighter-rouge">External Consistency</code> mode (see <a href="https://kudu.apache.org/docs/transaction_semantics.html">here</a>
for more information on how to use the relevant APIs).</p>
-<p>The first and most important knob, the <code>Read</code> mode, pertains to what is the guaranteed recency of
+<p>The first and most important knob, the <code class="highlighter-rouge">Read</code> mode, pertains to what is the guaranteed recency of
data resulting from scans. Since Kudu uses replication for availability and fault-tolerance, there
are always multiple replicas of any data item.
Not all replicas must be up-to-date so if the user cares about recency, e.g. if the user requires
that any data read includes all previously written data <em>from a single client instance</em> then it must
-choose the <code>READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe
+choose the <code class="highlighter-rouge">READ_AT_SNAPSHOT</code> read mode. With this mode enabled the client is guaranteed to observe
<strong>“READ YOUR OWN WRITES”</strong> semantics, i.e. scans from a client will always include all previous mutations
performed by that client. Note that this property is local to a single client instance, not a global
property.</p>
-<p>The second “knob”, the <code>External Consistency</code> mode, defines the semantics of how reads and writes
-are performed across multiple client instances. By default, <code>External Consistency</code> is set to
- <code>CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even
+<p>The second “knob”, the <code class="highlighter-rouge">External Consistency</code> mode, defines the semantics of how reads and writes
+are performed across multiple client instances. By default, <code class="highlighter-rouge">External Consistency</code> is set to
+ <code class="highlighter-rouge">CLIENT_PROPAGATED</code>, meaning it’s up to the user to coordinate a set of <em>timestamp tokens</em> with clients (even
across different machines) if they are performing writes/reads that are somehow causally linked.
If done correctly this enables <strong>STRICT SERIALIZABILITY</strong>[5], i.e. <strong>LINEARIZABILITY</strong>[6] and
<strong>SERIALIZABILITY</strong>[7] at the same time, at the cost of having the user coordinate the timestamp
tokens across clients (a survey of the meaning of these, and other definitions can be found
<a href="http://www.ics.forth.gr/tech-reports/2013/2013.TR439_Survey_on_Consistency_Conditions.pdf">here</a>).
-The alternative setting for <code>External Consistency</code> is to have it set to
-<code>COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by
+The alternative setting for <code class="highlighter-rouge">External Consistency</code> is to have it set to
+<code class="highlighter-rouge">COMMIT_WAIT</code> (experimental), which guarantees the same properties through a different means, by
implementing Google Spanner’s <em>TrueTime</em>. This comes at the cost of higher latency (depending on how
tightly synchronized the system clocks of the various tablet servers are), but doesn’t require users
to propagate timestamps programmatically.</p>
@@ -505,7 +608,7 @@ On the way, we’ll shed some light on the more relevant components and how they
fit together.
In Part 1 of the series (this one), we’ll cover motivation and design trade-offs, the end goals and
-the current status.</summary></entry><entry><title>Apache Kudu 1.5.0 released</title><link href="/2017/09/08/apache-kudu-1-5-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.5.0 released" /><published>2017-09-08T00:00:00+02:00</published><updated>2017-09-08T00:00:00+02:00</updated><id>/2017/09/08/apache-kudu-1-5-0-released</id><content type="html" xml:base="/2017/09/08/apache-kudu-1-5-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.5.0!</p>
+the current status.</summary></entry><entry><title>Apache Kudu 1.5.0 released</title><link href="/2017/09/08/apache-kudu-1-5-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.5.0 released" /><published>2017-09-08T00:00:00-07:00</published><updated>2017-09-08T00:00:00-07:00</updated><id>/2017/09/08/apache-kudu-1-5-0-released</id><content type="html" xml:base="/2017/09/08/apache-kudu-1-5-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.5.0!</p>
<p>Apache Kudu 1.5.0 is a minor release which offers several new features,
improvements, optimizations, and bug fixes.</p>
@@ -523,9 +626,9 @@ scenarios</li>
additional reductions planned for the future</li>
<li>a new configuration dashboard on the web UI which provides a high-level
summary of important configuration values</li>
- <li>a new <code>kudu tablet move</code> command which moves a tablet replica from one tablet
+ <li>a new <code class="highlighter-rouge">kudu tablet move</code> command which moves a tablet replica from one tablet
server to another</li>
- <li>a new <code>kudu local_replica data_size</code> command which summarizes the space usage
+ <li>a new <code class="highlighter-rouge">kudu local_replica data_size</code> command which summarizes the space usage
of a local tablet</li>
<li>all on-disk data is now checksummed by default, which provides error detection
for improved confidence when running Kudu on unreliable hardware</li>
@@ -546,7 +649,7 @@ repository.</li>
Apache Kudu 1.5.0 is a minor release which offers several new features,
improvements, optimizations, and bug fixes.
-Highlights include:</summary></entry><entry><title>Apache Kudu 1.4.0 released</title><link href="/2017/06/13/apache-kudu-1-4-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.4.0 released" /><published>2017-06-13T00:00:00+02:00</published><updated>2017-06-13T00:00:00+02:00</updated><id>/2017/06/13/apache-kudu-1-4-0-released</id><content type="html" xml:base="/2017/06/13/apache-kudu-1-4-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.4.0!</p>
+Highlights include:</summary></entry><entry><title>Apache Kudu 1.4.0 released</title><link href="/2017/06/13/apache-kudu-1-4-0-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.4.0 released" /><published>2017-06-13T00:00:00-07:00</published><updated>2017-06-13T00:00:00-07:00</updated><id>/2017/06/13/apache-kudu-1-4-0-released</id><content type="html" xml:base="/2017/06/13/apache-kudu-1-4-0-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.4.0!</p>
<p>Apache Kudu 1.4.0 is a minor release which offers several new features,
improvements, optimizations, and bug fixes.</p>
@@ -560,7 +663,7 @@ improvements, optimizations, and bug fixes.</p>
<li>a new C++ client API to efficiently map primary keys to their associated partitions
and hosts</li>
<li>support for long-running fault-tolerant scans in the Java client</li>
- <li>a new <code>kudu fs check</code> command which can perform offline consistency checks
+ <li>a new <code class="highlighter-rouge">kudu fs check</code> command which can perform offline consistency checks
and repairs on the local on-disk storage of a Tablet Server or Master.</li>
<li>many optimizations to reduce disk space usage, improve write throughput,
and improve throughput of background maintenance operations.</li>
@@ -581,31 +684,4 @@ repository.</li>
Apache Kudu 1.4.0 is a minor release which offers several new features,
improvements, optimizations, and bug fixes.
-Highlights include:</summary></entry><entry><title>Apache Kudu 1.3.1 released</title><link href="/2017/04/19/apache-kudu-1-3-1-released.html" rel="alternate" type="text/html" title="Apache Kudu 1.3.1 released" /><published>2017-04-19T00:00:00+02:00</published><updated>2017-04-19T00:00:00+02:00</updated><id>/2017/04/19/apache-kudu-1-3-1-released</id><content type="html" xml:base="/2017/04/19/apache-kudu-1-3-1-released.html"><p>The Apache Kudu team is happy to announce the release of Kudu 1.3.1!</p>
-
-<p>Apache Kudu 1.3.1 is a bug fix release which fixes critical issues discovered
-in Apache Kudu 1.3.0. In particular, this fixes a bug in which data could be
-incorrectly deleted after certain sequences of node failures. Several other
-bugs are also fixed. See the release notes for details.</p>
-
-<p>Users of Kudu 1.3.0 are encouraged to upgrade to 1.3.1 immediately.</p>
-
-<ul>
- <li>Download the <a href="/releases/1.3.1/">Kudu 1.3.1 source release</a></li>
- <li>Convenience binary artifacts for the Java client and various Java
-integrations (eg Spark, Flume) are also now available via the ASF Maven
-repository.</li>
-</ul></content><author><name>Todd Lipcon</name></author><summary>The Apache Kudu team is happy to announce the release of Kudu 1.3.1!
-
-Apache Kudu 1.3.1 is a bug fix release which fixes critical issues discovered
-in Apache Kudu 1.3.0. In particular, this fixes a bug in which data could be
-incorrectly deleted after certain sequences of node failures. Several other
-bugs are also fixed. See the release notes for details.
-
-Users of Kudu 1.3.0 are encouraged to upgrade to 1.3.1 immediately.
-
-
- Download the Kudu 1.3.1 source release
- Convenience binary artifacts for the Java client and various Java
-integrations (eg Spark, Flume) are also now available via the ASF Maven
-repository.</summary></entry></feed>
+Highlights include:</summary></entry></feed>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/img/index-skip-scan/example-table.png
----------------------------------------------------------------------
diff --git a/img/index-skip-scan/example-table.png b/img/index-skip-scan/example-table.png
new file mode 100644
index 0000000..585ae4d
Binary files /dev/null and b/img/index-skip-scan/example-table.png differ
[4/4] kudu-site git commit: Publish commit(s) from site source repo:
83530755d Blogpost describing index skip scan optimization.
Posted by mp...@apache.org.
Publish commit(s) from site source repo:
83530755d Blogpost describing index skip scan optimization.
Site-Repo-Commit: 83530755dbcddd44debe9b3ade286f58b96d5c82
Project: http://git-wip-us.apache.org/repos/asf/kudu-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu-site/commit/12782cec
Tree: http://git-wip-us.apache.org/repos/asf/kudu-site/tree/12782cec
Diff: http://git-wip-us.apache.org/repos/asf/kudu-site/diff/12782cec
Branch: refs/heads/asf-site
Commit: 12782cec3b2222d1e12c4c2f17ffc83996944221
Parents: 8efb67b
Author: Mike Percy <mp...@cloudera.com>
Authored: Wed Sep 26 10:55:47 2018 -0700
Committer: Mike Percy <mp...@cloudera.com>
Committed: Wed Sep 26 10:55:47 2018 -0700
----------------------------------------------------------------------
2016/02/26/apache-kudu-0-7-0-released.html | 6 +-
2016/03/10/apache-kudu-0-7-1-released.html | 4 +-
2016/03/20/weekly-update.html | 4 +-
2016/03/28/weekly-update.html | 4 +-
2016/04/04/weekly-update.html | 4 +-
2016/04/11/weekly-update.html | 4 +-
2016/04/12/apache-kudu-0-8-0-released.html | 4 +-
...n-apache-kudu-streamsets-data-collector.html | 4 +-
2016/04/18/weekly-update.html | 4 +-
.../19/kudu-0-8-0-predicate-improvements.html | 16 +-
2016/04/25/weekly-update.html | 4 +-
2016/04/26/ycsb.html | 80 ++---
2016/05/03/weekly-update.html | 4 +-
2016/05/09/weekly-update.html | 4 +-
2016/05/16/weekly-update.html | 4 +-
2016/05/23/weekly-update.html | 10 +-
2016/06/01/weekly-update.html | 4 +-
2016/06/02/no-default-partitioning.html | 14 +-
2016/06/06/weekly-update.html | 4 +-
2016/06/10/apache-kudu-0-9-0-released.html | 4 +-
2016/06/13/weekly-update.html | 4 +-
2016/06/17/raft-consensus-single-node.html | 8 +-
2016/06/21/weekly-update.html | 6 +-
2016/06/24/multi-master-1-0-0.html | 10 +-
2016/06/27/weekly-update.html | 4 +-
2016/07/01/apache-kudu-0-9-1-released.html | 4 +-
2016/07/11/weekly-update.html | 4 +-
2016/07/18/weekly-update.html | 4 +-
2016/07/25/asf-graduation.html | 4 +-
2016/07/26/weekly-update.html | 20 +-
2016/08/08/weekly-update.html | 4 +-
2016/08/16/weekly-update.html | 18 +-
2016/08/23/apache-kudu-0-10-0-released.html | 4 +-
2016/08/23/new-range-partitioning-features.html | 4 +-
2016/08/31/intro-flume-kudu-sink.html | 160 ++++-----
2016/09/16/predicate-pushdown.html | 4 +-
2016/09/20/apache-kudu-1-0-0-released.html | 10 +-
2016/09/26/strata-nyc-kudu-talks.html | 4 +-
2016/10/11/weekly-update.html | 14 +-
2016/10/20/weekly-update.html | 4 +-
2016/11/01/weekly-update.html | 4 +-
2016/11/15/weekly-update.html | 4 +-
2017/01/20/apache-kudu-1-2-0-released.html | 4 +-
2017/03/20/apache-kudu-1-3-0-released.html | 4 +-
2017/04/19/apache-kudu-1-3-1-released.html | 4 +-
2017/06/13/apache-kudu-1-4-0-released.html | 6 +-
2017/09/08/apache-kudu-1-5-0-released.html | 8 +-
2017/09/18/kudu-consistency-pt1.html | 20 +-
2017/10/23/nosql-kudu-spanner-slides.html | 4 +-
2017/12/08/apache-kudu-1-6-0-released.html | 4 +-
2018/03/23/apache-kudu-1-7-0-released.html | 4 +-
2018/07/10/instrumentation-in-kudu.html | 4 +-
...ting-started-with-kudu-an-oreilly-title.html | 10 +-
2018/09/11/simplified-pipelines-with-kudu.html | 4 +-
.../index-skip-scan-optimization-in-kudu.html | 320 +++++++++++++++++
blog/index.html | 52 ++-
blog/page/10/index.html | 48 +--
blog/page/11/index.html | 25 +-
blog/page/2/index.html | 63 ++--
blog/page/3/index.html | 59 ++--
blog/page/4/index.html | 339 ++-----------------
blog/page/5/index.html | 339 +++++++++++++++++--
blog/page/6/index.html | 46 +--
blog/page/7/index.html | 56 ++-
blog/page/8/index.html | 56 +--
blog/page/9/index.html | 48 ++-
faq.html | 12 +-
feed.xml | 178 +++++++---
img/index-skip-scan/example-table.png | Bin 0 -> 73625 bytes
img/index-skip-scan/skip-scan-example-table.png | Bin 0 -> 48512 bytes
.../skip-scan-performance-graph.png | Bin 0 -> 59142 bytes
71 files changed, 1312 insertions(+), 893 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/02/26/apache-kudu-0-7-0-released.html
----------------------------------------------------------------------
diff --git a/2016/02/26/apache-kudu-0-7-0-released.html b/2016/02/26/apache-kudu-0-7-0-released.html
index eccf5fc..1f1d7b5 100644
--- a/2016/02/26/apache-kudu-0-7-0-released.html
+++ b/2016/02/26/apache-kudu-0-7-0-released.html
@@ -126,7 +126,7 @@ part of the ASF Incubator, version 0.7.0!</p>
<!--more-->
<ul>
- <li>Kudu now has limited support for Apache Spark, through a simple <code>kuduRDD</code> API</li>
+ <li>Kudu now has limited support for Apache Spark, through a simple <code class="highlighter-rouge">kuduRDD</code> API</li>
<li>Several improvements to the Java client including better error handling support and fixes for a potential client hang.</li>
<li>Better platform support, including support for building on RHEL 7, CentOS 7, and SLES 12.</li>
<li>The Kudu Python client has been completely revamped and is now considered a supported feature.</li>
@@ -149,6 +149,8 @@ part of the ASF Incubator, version 0.7.0!</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -177,8 +179,6 @@ part of the ASF Incubator, version 0.7.0!</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/03/10/apache-kudu-0-7-1-released.html
----------------------------------------------------------------------
diff --git a/2016/03/10/apache-kudu-0-7-1-released.html b/2016/03/10/apache-kudu-0-7-1-released.html
index 5f406f3..f1eeed8 100644
--- a/2016/03/10/apache-kudu-0-7-1-released.html
+++ b/2016/03/10/apache-kudu-0-7-1-released.html
@@ -148,6 +148,8 @@ bugs fixed in this release:
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -176,8 +178,6 @@ bugs fixed in this release:
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/03/20/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/03/20/weekly-update.html b/2016/03/20/weekly-update.html
index 7b562ea..2e00f02 100644
--- a/2016/03/20/weekly-update.html
+++ b/2016/03/20/weekly-update.html
@@ -208,6 +208,8 @@ at the <a href="http://www.meetup.com/Rust-Bay-Area/events/229107276/">Rust Bay
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -236,8 +238,6 @@ at the <a href="http://www.meetup.com/Rust-Bay-Area/events/229107276/">Rust Bay
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/03/28/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/03/28/weekly-update.html b/2016/03/28/weekly-update.html
index 04d2bb7..54e625c 100644
--- a/2016/03/28/weekly-update.html
+++ b/2016/03/28/weekly-update.html
@@ -199,6 +199,8 @@ in San Jose. The conference will feature two talks on Kudu:</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -227,8 +229,6 @@ in San Jose. The conference will feature two talks on Kudu:</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/04/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/04/04/weekly-update.html b/2016/04/04/weekly-update.html
index 1a77942..ff1ec3c 100644
--- a/2016/04/04/weekly-update.html
+++ b/2016/04/04/weekly-update.html
@@ -218,6 +218,8 @@ on Friday, April 8th.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -246,8 +248,6 @@ on Friday, April 8th.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/11/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/04/11/weekly-update.html b/2016/04/11/weekly-update.html
index 149385a..663ca93 100644
--- a/2016/04/11/weekly-update.html
+++ b/2016/04/11/weekly-update.html
@@ -218,6 +218,8 @@ please pipe up on the mailing list.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -246,8 +248,6 @@ please pipe up on the mailing list.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/12/apache-kudu-0-8-0-released.html
----------------------------------------------------------------------
diff --git a/2016/04/12/apache-kudu-0-8-0-released.html b/2016/04/12/apache-kudu-0-8-0-released.html
index d388388..f5254c2 100644
--- a/2016/04/12/apache-kudu-0-8-0-released.html
+++ b/2016/04/12/apache-kudu-0-8-0-released.html
@@ -139,6 +139,8 @@ client, plus many other improvements and bug fixes.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -167,8 +169,6 @@ client, plus many other improvements and bug fixes.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
----------------------------------------------------------------------
diff --git a/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html b/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
index 0e18356..447da2e 100644
--- a/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
+++ b/2016/04/14/ingesting-json-apache-kudu-streamsets-data-collector.html
@@ -162,6 +162,8 @@ kernel drivers in C to web front ends in JavaScript.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -190,8 +192,6 @@ kernel drivers in C to web front ends in JavaScript.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/18/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/04/18/weekly-update.html b/2016/04/18/weekly-update.html
index 1c866f2..17607de 100644
--- a/2016/04/18/weekly-update.html
+++ b/2016/04/18/weekly-update.html
@@ -216,6 +216,8 @@ Data Collector</a>.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -244,8 +246,6 @@ Data Collector</a>.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/19/kudu-0-8-0-predicate-improvements.html
----------------------------------------------------------------------
diff --git a/2016/04/19/kudu-0-8-0-predicate-improvements.html b/2016/04/19/kudu-0-8-0-predicate-improvements.html
index 7af1350..aa0742b 100644
--- a/2016/04/19/kudu-0-8-0-predicate-improvements.html
+++ b/2016/04/19/kudu-0-8-0-predicate-improvements.html
@@ -157,10 +157,10 @@ know on the <a href="https://issues.apache.org/jira/browse/KUDU-1065">JIRA issue
<h2 id="redesigned-predicate-api-in-the-java-client">Redesigned Predicate API in the Java Client</h2>
<p>The Java client has a new way to express scan predicates: the
-<a href="http://kudu.apache.org/apidocs/org/kududb/client/KuduPredicate.html"><code>KuduPredicate</code></a>.
+<a href="http://kudu.apache.org/apidocs/org/kududb/client/KuduPredicate.html"><code class="highlighter-rouge">KuduPredicate</code></a>.
The API matches the corresponding C++ API more closely, and adds support for
specifying exclusive, as well as inclusive, range predicates. The existing
-<a href="http://kudu.apache.org/apidocs/org/kududb/client/ColumnRangePredicate.html"><code>ColumnRangePredicate</code></a>
+<a href="http://kudu.apache.org/apidocs/org/kududb/client/ColumnRangePredicate.html"><code class="highlighter-rouge">ColumnRangePredicate</code></a>
API has been deprecated, and will be removed soon. Example of transitioning from
the old to new API:</p>
@@ -178,14 +178,14 @@ the old to new API:</p>
<h2 id="under-the-covers-changes">Under the Covers Changes</h2>
-<p>The scan optimizations in the server and C++ client, and the new <code>KuduPredicate</code>
+<p>The scan optimizations in the server and C++ client, and the new <code class="highlighter-rouge">KuduPredicate</code>
API in the Java client are made possible by an overhaul of how predicates are
handled internally. A new protobuf message type,
-<a href="https://github.com/apache/incubator-kudu/blob/master/src/kudu/common/common.proto#L273"><code>ColumnPredicatePB</code></a>
+<a href="https://github.com/apache/incubator-kudu/blob/master/src/kudu/common/common.proto#L273"><code class="highlighter-rouge">ColumnPredicatePB</code></a>
has been introduced, and will allow more column predicate types to be introduced
in the future. If you are interested in contributing to Kudu but don’t know
-where to start, consider adding a new predicate type; for example the <code>IS NULL</code>,
-<code>IS NOT NULL</code>, <code>IN</code>, and <code>LIKE</code> predicates types are currently not implemented.</p>
+where to start, consider adding a new predicate type; for example the <code class="highlighter-rouge">IS NULL</code>,
+<code class="highlighter-rouge">IS NOT NULL</code>, <code class="highlighter-rouge">IN</code>, and <code class="highlighter-rouge">LIKE</code> predicates types are currently not implemented.</p>
</div>
</article>
@@ -196,6 +196,8 @@ where to start, consider adding a new predicate type; for example the <code>IS N
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -224,8 +226,6 @@ where to start, consider adding a new predicate type; for example the <code>IS N
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/25/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/04/25/weekly-update.html b/2016/04/25/weekly-update.html
index 4fc73d9..fa75dea 100644
--- a/2016/04/25/weekly-update.html
+++ b/2016/04/25/weekly-update.html
@@ -215,6 +215,8 @@ Kudu 0.8</a>.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -243,8 +245,6 @@ Kudu 0.8</a>.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/04/26/ycsb.html
----------------------------------------------------------------------
diff --git a/2016/04/26/ycsb.html b/2016/04/26/ycsb.html
index 2432c17..6d549d7 100644
--- a/2016/04/26/ycsb.html
+++ b/2016/04/26/ycsb.html
@@ -145,12 +145,12 @@
</ul>
<h2 id="experimental-setup">Experimental setup</h2>
-<p>The single-node Kudu cluster was configured, started, and stopped by a Python script <code>run_experiments.py</code> which cycled through several different configurations, completely removing all data in between each iteration. For each Kudu configuration, YCSB was used to load 100M rows of data (each approximately 1KB). YCSB is configured with 16 client threads on the same node. For each configuration, the YCSB log as well as periodic dumps of Tablet Server metrics are captured for later analysis.</p>
+<p>The single-node Kudu cluster was configured, started, and stopped by a Python script <code class="highlighter-rouge">run_experiments.py</code> which cycled through several different configurations, completely removing all data in between each iteration. For each Kudu configuration, YCSB was used to load 100M rows of data (each approximately 1KB). YCSB is configured with 16 client threads on the same node. For each configuration, the YCSB log as well as periodic dumps of Tablet Server metrics are captured for later analysis.</p>
<p>Note that in many cases, the 16 client threads were not enough to max out the full performance of the machine. These experiments should not be taken to determine the maximum throughput of Kudu – instead, we are looking at comparing the <em>relative</em> performance of different configuration options.</p>
<h1 id="benchmarking-synchronous-insert-operations">Benchmarking Synchronous Insert Operations</h1>
-<p>The first set of experiments runs the YCSB load with the <code>sync_ops=true</code> configuration option. This option means that each client thread will insert one row at a time and synchronously wait for the response before inserting the next row. The lack of batching makes this a good stress test for Kudu’s RPC performance and other fixed per-request costs.</p>
+<p>The first set of experiments runs the YCSB load with the <code class="highlighter-rouge">sync_ops=true</code> configuration option. This option means that each client thread will insert one row at a time and synchronously wait for the response before inserting the next row. The lack of batching makes this a good stress test for Kudu’s RPC performance and other fixed per-request costs.</p>
<p>The fact that the requests are synchronous also makes it easy to measure the <em>latency</em> of the write requests. With request batching enabled, latency would be irrelevant.</p>
@@ -170,25 +170,25 @@
<p><img src="/img/YCSB_files/YCSB_3_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 31163 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 31163 ops/sec
+</div>
<p>The results here are interesting: the throughput starts out around 70K rows/second, but then collapses to nearly zero. After staying near zero for a while, it shoots back up to the original performance, and the pattern repeats many times.</p>
<p>Also note that the 99th percentile latency seems to alternate between close to zero and a value near 500ms. This bimodal distribution led me to grep in the Java source for the magic number 500. Sure enough, I found:</p>
-<pre><code class="language-java">public static final int SLEEP_TIME = 500;
-</code></pre>
+<div class="highlighter-rouge"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">SLEEP_TIME</span> <span class="o">=</span> <span class="mi">500</span><span class="o">;</span>
+</div>
<p>Used in this backoff calculation method (slightly paraphrased here):</p>
-<pre><code class="language-java"> long getSleepTimeForRpc(KuduRpc<?> rpc) {
- // TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50
- return (attemptCount * SLEEP_TIME) + sleepRandomizer.nextInt(50);
- }
-</code></pre>
+<div class="highlighter-rouge"> <span class="kt">long</span> <span class="nf">getSleepTimeForRpc</span><span class="o">(</span><span class="n">KuduRpc</span><span class="o"><?></span> <span class="n">rpc</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// TODO backoffs? Sleep in increments of 500 ms, plus some random time up to 50</span>
+ <span class="k">return</span> <span class="o">(</span><span class="n">attemptCount</span> <span class="o">*</span> <span class="n">SLEEP_TIME</span><span class="o">)</span> <span class="o">+</span> <span class="n">sleepRandomizer</span><span class="o">.</span><span class="na">nextInt</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="o">}</span>
+</div>
-<p>One reason that a client will back off and retry is a <code>SERVER_TOO_BUSY</code> response from the server. This response is used in a number of overload situations. In a write-mostly workload, the most likely situation is that the server is low on memory and thus asking clients to back off while it flushes. Sure enough, when we graph the heap usage over time, as well as the rate of writes rejected due to low-memory, we see that this is the case:</p>
+<p>One reason that a client will back off and retry is a <code class="highlighter-rouge">SERVER_TOO_BUSY</code> response from the server. This response is used in a number of overload situations. In a write-mostly workload, the most likely situation is that the server is low on memory and thus asking clients to back off while it flushes. Sure enough, when we graph the heap usage over time, as well as the rate of writes rejected due to low-memory, we see that this is the case:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"heap_allocated"</span><span class="p">,</span> <span class="s">"Heap usage (GB)"</span><span class="p">,</span> <span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">)</span>
<span class="n">plot_ts_metric</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'default'</span><span class="p">],</span> <span class="s">"mem_rejections"</span><span class="p">,</span> <span class="s">"Rejected writes</span><span class="se">\n</span><span class="s">per sec"</span><span class="p">)</span></code></pre></div>
@@ -205,24 +205,24 @@
<p><img src="/img/YCSB_files/YCSB_7_0.png" alt="png" class="img-responsive" /></p>
-<p>I then re-ran the workload while watching <code>iostat -dxm 1</code> to see the write rates across all of the disks. I could see that each of the disks was busy in turn, rather than busy in parallel.</p>
+<p>I then re-ran the workload while watching <code class="highlighter-rouge">iostat -dxm 1</code> to see the write rates across all of the disks. I could see that each of the disks was busy in turn, rather than busy in parallel.</p>
<p>This reminded me that the default way in which Kudu flushes data is as follows:</p>
-<pre><code>for each column:
+<div class="highlighter-rouge">for each column:
open a new block on disk to write that column, round-robining across disks
iterate over data:
append data to the already-open blocks
for each column:
fsync() the block of data
close the block
-</code></pre>
+</div>
-<p>Because Kudu uses buffered writes, the actual appending of data to the open blocks does not generate immediate IO. Instead, it only dirties pages in the Linux page cache. The actual IO is performed with the <code>fsync</code> call at the end. Because Kudu defaults to fsyncing each file in turn from a single thread, this was causing the slow performance identified above.</p>
+<p>Because Kudu uses buffered writes, the actual appending of data to the open blocks does not generate immediate IO. Instead, it only dirties pages in the Linux page cache. The actual IO is performed with the <code class="highlighter-rouge">fsync</code> call at the end. Because Kudu defaults to fsyncing each file in turn from a single thread, this was causing the slow performance identified above.</p>
-<p>At this point, I consulted with Adar Dembo, who designed much of this code path. He reminded me that we actually have a configuration flag <code>cfile_do_on_finish=flush</code> which changes the code to something resembling the following:</p>
+<p>At this point, I consulted with Adar Dembo, who designed much of this code path. He reminded me that we actually have a configuration flag <code class="highlighter-rouge">cfile_do_on_finish=flush</code> which changes the code to something resembling the following:</p>
-<pre><code>for each column:
+<div class="highlighter-rouge">for each column:
open a new block on disk to write that column, round-robining across disks
iterate over data:
append data to the already-open blocks
@@ -231,16 +231,16 @@ for each column:
for each column:
fsync the block
close the block
-</code></pre>
+</div>
-<p>The <code>sync_file_range</code> call here asynchronously enqueues the dirty pages to be written back to the disks, and then the following <code>fsync</code> actually waits for the writeback to be complete. I ran the benchmark for a new configuration with this flag enabled, and plotted the results:</p>
+<p>The <code class="highlighter-rouge">sync_file_range</code> call here asynchronously enqueues the dirty pages to be written back to the disks, and then the following <code class="highlighter-rouge">fsync</code> actually waits for the writeback to be complete. I ran the benchmark for a new configuration with this flag enabled, and plotted the results:</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush'</span><span class="p">])</span></code></pre></div>
<p><img src="/img/YCSB_files/YCSB_9_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 52457 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 52457 ops/sec
+</div>
<p>This is already a substantial improvement from the default settings. The overall throughput has increased from 31K ops/second to 52K ops/second (<strong>67%</strong>), and we no longer see any dramatic drops in performance or increases in 99th percentile. In fact, the 99th percentile stays comfortably below 1ms for the entire test.</p>
@@ -282,14 +282,14 @@ for each column:
<p>Writing a lot of small flushes compared to a small number of large flushes means that the on-disk data is not as well sorted in the optimized workload. An individual write may need to consult up to 20 bloom filters corresponding to previously flushed pieces of data in order to ensure that it is not an insert with a duplicate primary key.</p>
-<p>So, how can we address this issue? It turns out that the flush threshold is actually configurable with the <code>flush_threshold_mb</code> flag. I re-ran the workload yet another time with the flush threshold set to 20GB.</p>
+<p>So, how can we address this issue? It turns out that the flush threshold is actually configurable with the <code class="highlighter-rouge">flush_threshold_mb</code> flag. I re-ran the workload yet another time with the flush threshold set to 20GB.</p>
<div class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">plot_throughput_latency</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s">'finish=flush+20GB-threshold'</span><span class="p">])</span></code></pre></div>
<p><img src="/img/YCSB_files/YCSB_17_0.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 67123 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 67123 ops/sec
+</div>
<p>This gets us another 28% improvement from 52K ops/second up to 67K ops/second (<strong>+116%</strong> from the default), and we no longer see the troubling downward slope on the throughput graph. Let’s check on the memory and bloom filter metrics again.</p>
@@ -318,13 +318,13 @@ for each column:
<h1 id="tests-with-batched-writes">Tests with Batched Writes</h1>
-<p>The above tests were done with the <code>sync_ops=true</code> YCSB configuration option. However, we expect that for many heavy write situations, the writers would batch many rows together into larger write operations for better throughput.</p>
+<p>The above tests were done with the <code class="highlighter-rouge">sync_ops=true</code> YCSB configuration option. However, we expect that for many heavy write situations, the writers would batch many rows together into larger write operations for better throughput.</p>
-<p>I wanted to ensure that the recommended configuration changes above also improved performance for this workload. So, I re-ran the same experiments, but with YCSB configured to send batches of 100 insert operations to the tablet server using the Kudu client’s <code>AUTO_FLUSH_BACKGROUND</code> write mode.</p>
+<p>I wanted to ensure that the recommended configuration changes above also improved performance for this workload. So, I re-ran the same experiments, but with YCSB configured to send batches of 100 insert operations to the tablet server using the Kudu client’s <code class="highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write mode.</p>
<p>This time, I compared four configurations:
- the Kudu default settings
-- the defaults, but configured with <code>cfile_do_on_finish=flush</code> to increase flush IO performance
+- the defaults, but configured with <code class="highlighter-rouge">cfile_do_on_finish=flush</code> to increase flush IO performance
- the above, but with the flush thresholds configured to 1G and 10G</p>
<p>For these experiments, we don’t plot latencies, since write latencies are meaningless with batching enabled.</p>
@@ -341,8 +341,8 @@ for each column:
<p><img src="/img/YCSB_files/YCSB_23_1.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 33319 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 33319 ops/sec
+</div>
<p><img src="/img/YCSB_files/YCSB_23_3.png" alt="png" class="img-responsive" /></p>
@@ -351,8 +351,8 @@ for each column:
<p><img src="/img/YCSB_files/YCSB_23_5.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 80068 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 80068 ops/sec
+</div>
<p><img src="/img/YCSB_files/YCSB_23_7.png" alt="png" class="img-responsive" /></p>
@@ -361,8 +361,8 @@ for each column:
<p><img src="/img/YCSB_files/YCSB_23_9.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 78040 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 78040 ops/sec
+</div>
<p><img src="/img/YCSB_files/YCSB_23_11.png" alt="png" class="img-responsive" /></p>
@@ -371,8 +371,8 @@ for each column:
<p><img src="/img/YCSB_files/YCSB_23_13.png" alt="png" class="img-responsive" /></p>
-<pre><code>Average throughput: 82005 ops/sec
-</code></pre>
+<div class="highlighter-rouge">Average throughput: 82005 ops/sec
+</div>
<p><img src="/img/YCSB_files/YCSB_23_15.png" alt="png" class="img-responsive" /></p>
@@ -393,8 +393,8 @@ for each column:
<p>We will likely make these changes in the next Kudu release. In the meantime, users can experiment by adding the following flags to their tablet server configuration:</p>
<ul>
- <li><code>--cfile_do_on_finish=flush</code></li>
- <li><code>--flush_threshold_mb=10000</code></li>
+ <li><code class="highlighter-rouge">--cfile_do_on_finish=flush</code></li>
+ <li><code class="highlighter-rouge">--flush_threshold_mb=10000</code></li>
</ul>
<p>Note that, even if the server hosts many tablets or has less memory than the one used in this test, flushes will still be triggered if the <em>overall</em> memory consumption of the process crosses the configured soft limit. So, configuring a 10GB threshold does not increase the risk of out-of-memory errors.</p>
@@ -418,6 +418,8 @@ for each column:
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -446,8 +448,6 @@ for each column:
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/05/03/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/05/03/weekly-update.html b/2016/05/03/weekly-update.html
index 106b692..9f5540a 100644
--- a/2016/05/03/weekly-update.html
+++ b/2016/05/03/weekly-update.html
@@ -190,6 +190,8 @@ list of conferenace sessions and meetups near you.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -218,8 +220,6 @@ list of conferenace sessions and meetups near you.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/05/09/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/05/09/weekly-update.html b/2016/05/09/weekly-update.html
index 650c081..77d2fb5 100644
--- a/2016/05/09/weekly-update.html
+++ b/2016/05/09/weekly-update.html
@@ -180,6 +180,8 @@ on May 10.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -208,8 +210,6 @@ on May 10.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/05/16/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/05/16/weekly-update.html b/2016/05/16/weekly-update.html
index 5f2e1bd..7fe1a1f 100644
--- a/2016/05/16/weekly-update.html
+++ b/2016/05/16/weekly-update.html
@@ -215,6 +215,8 @@ meetup.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -243,8 +245,6 @@ meetup.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/05/23/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/05/23/weekly-update.html b/2016/05/23/weekly-update.html
index 4ce4bcc..d4259e9 100644
--- a/2016/05/23/weekly-update.html
+++ b/2016/05/23/weekly-update.html
@@ -149,9 +149,9 @@ first release candidate.</p>
</li>
<li>
<p>Since Kudu’s initial release, one of the most commonly requested features
-has been support for the <code>UPSERT</code> operation. <code>UPSERT</code>, known in some other
-databases as <code>INSERT ... ON DUPLICATE KEY UPDATE</code>. This operation has the
-semantics of an <code>INSERT</code> if no key already exists with the provided primary
+has been support for the <code class="highlighter-rouge">UPSERT</code> operation. <code class="highlighter-rouge">UPSERT</code>, known in some other
+databases as <code class="highlighter-rouge">INSERT ... ON DUPLICATE KEY UPDATE</code>. This operation has the
+semantics of an <code class="highlighter-rouge">INSERT</code> if no key already exists with the provided primary
key. Otherwise, it replaces the existing row with the new values.</p>
<p>This week, several developers collaborated to add support for this operation.
@@ -204,6 +204,8 @@ Cloudera User Group</a>.</li>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -232,8 +234,6 @@ Cloudera User Group</a>.</li>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/01/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/06/01/weekly-update.html b/2016/06/01/weekly-update.html
index bc40a1e..40fbd88 100644
--- a/2016/06/01/weekly-update.html
+++ b/2016/06/01/weekly-update.html
@@ -170,6 +170,8 @@ hadoop-common test jar. This solved build issues while also removing a nasty dep
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -198,8 +200,6 @@ hadoop-common test jar. This solved build issues while also removing a nasty dep
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/02/no-default-partitioning.html
----------------------------------------------------------------------
diff --git a/2016/06/02/no-default-partitioning.html b/2016/06/02/no-default-partitioning.html
index c723657..2e3a9c9 100644
--- a/2016/06/02/no-default-partitioning.html
+++ b/2016/06/02/no-default-partitioning.html
@@ -161,7 +161,7 @@ advanced configurations.</p>
<h1 id="c-client">C++ Client</h1>
<p>With the C++ client, creating a new table with hash partitions is as simple as
-calling <code>KuduTableCreator:add_hash_partitions</code> with the columns to hash and the
+calling <code class="highlighter-rouge">KuduTableCreator:add_hash_partitions</code> with the columns to hash and the
number of buckets to use:</p>
<div class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">unique_ptr</span><span class="o"><</span><span class="n">KuduTableCreator</span><span class="o">></span> <span class="n">table_creator</span><span class="p">(</span><span class="n">my_client</span><span class="o">-></span><span class="n">NewTableCreator</span><span class="p">());</span>
@@ -182,14 +182,14 @@ number of buckets to use:</p>
<span class="n">myClient</span><span class="o">.</span><span class="na">createTable</span><span class="o">(</span><span class="s">"my-table"</span><span class="o">,</span> <span class="n">my_schema</span><span class="o">,</span> <span class="n">options</span><span class="o">);</span></code></pre></div>
<p>In the examples above, if the hash partition configuration is omitted the create
-table operation will fail with the error <code>Table partitioning must be specified
+table operation will fail with the error <code class="highlighter-rouge">Table partitioning must be specified
using setRangePartitionColumns or addHashPartitions</code>. In the Java client this
-manifests as a thrown <code>IllegalArgumentException</code>, while in the C++ client it is
-returned as a <code>Status::InvalidArgument</code>.</p>
+manifests as a thrown <code class="highlighter-rouge">IllegalArgumentException</code>, while in the C++ client it is
+returned as a <code class="highlighter-rouge">Status::InvalidArgument</code>.</p>
<h1 id="impala">Impala</h1>
-<p>When creating Kudu tables with Impala, the formerly optional <code>DISTRIBUTE BY</code>
+<p>When creating Kudu tables with Impala, the formerly optional <code class="highlighter-rouge">DISTRIBUTE BY</code>
clause is now required:</p>
<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span><span class="n">key_column_a</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">key_column_b</span> <span class="n">STRING</span><span class="p">,</span> <span class="n">other_column</span> <span class="n">STRING</span><span class="p">)</span>
@@ -211,6 +211,8 @@ clause is now required:</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -239,8 +241,6 @@ clause is now required:</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/06/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/06/06/weekly-update.html b/2016/06/06/weekly-update.html
index 57e9c60..34105e3 100644
--- a/2016/06/06/weekly-update.html
+++ b/2016/06/06/weekly-update.html
@@ -165,6 +165,8 @@ patches in for the <a href="https://gerrit.cloudera.org/#/c/2642/">Replay Cache<
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -193,8 +195,6 @@ patches in for the <a href="https://gerrit.cloudera.org/#/c/2642/">Replay Cache<
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/10/apache-kudu-0-9-0-released.html
----------------------------------------------------------------------
diff --git a/2016/06/10/apache-kudu-0-9-0-released.html b/2016/06/10/apache-kudu-0-9-0-released.html
index 635e4d0..00bd3b1 100644
--- a/2016/06/10/apache-kudu-0-9-0-released.html
+++ b/2016/06/10/apache-kudu-0-9-0-released.html
@@ -140,6 +140,8 @@ the specification of a partitioning scheme for new tables.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -168,8 +170,6 @@ the specification of a partitioning scheme for new tables.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/13/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/06/13/weekly-update.html b/2016/06/13/weekly-update.html
index c3b21dc..b051ffa 100644
--- a/2016/06/13/weekly-update.html
+++ b/2016/06/13/weekly-update.html
@@ -173,6 +173,8 @@ removal happening in this <a href="https://gerrit.cloudera.org/#/c/3350/">patch<
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -201,8 +203,6 @@ removal happening in this <a href="https://gerrit.cloudera.org/#/c/3350/">patch<
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/17/raft-consensus-single-node.html
----------------------------------------------------------------------
diff --git a/2016/06/17/raft-consensus-single-node.html b/2016/06/17/raft-consensus-single-node.html
index ded4092..6c532c7 100644
--- a/2016/06/17/raft-consensus-single-node.html
+++ b/2016/06/17/raft-consensus-single-node.html
@@ -143,13 +143,13 @@ implementation was complete.</p>
<p>The Consensus API has the following main responsibilities:</p>
<ol>
- <li>Support acting as a Raft <code>LEADER</code> and replicate writes to a local
+ <li>Support acting as a Raft <code class="highlighter-rouge">LEADER</code> and replicate writes to a local
write-ahead log (WAL) as well as followers in the Raft configuration. For
each operation written to the leader, a Raft implementation must keep track
of how many nodes have written a copy of the operation being replicated, and
whether or not that constitutes a majority. Once a majority of the nodes
have written a copy of the data, it is considered committed.</li>
- <li>Support acting as a Raft <code>FOLLOWER</code> by accepting writes from the leader and
+ <li>Support acting as a Raft <code class="highlighter-rouge">FOLLOWER</code> by accepting writes from the leader and
preparing them to be eventually committed.</li>
<li>Support voting in and initiating leader elections.</li>
<li>Support participating in and initiating configuration changes (such as going
@@ -215,6 +215,8 @@ dissertation, which you can find linked from the above web site.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -243,8 +245,6 @@ dissertation, which you can find linked from the above web site.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/21/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/06/21/weekly-update.html b/2016/06/21/weekly-update.html
index 51f557f..e152b57 100644
--- a/2016/06/21/weekly-update.html
+++ b/2016/06/21/weekly-update.html
@@ -134,7 +134,7 @@ leveraging the tablets cache.</p>
<li>
<p>In the context of making multi-master reliable in 1.0, Adar Dembo posted a <a href="https://gerrit.cloudera.org/#/c/3393/">design document</a>
on how to handle permanent master failures. Currently the master’s code is missing some features
-like <code>remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data
+like <code class="highlighter-rouge">remote bootstrap</code> which makes it possible for a new replica to download a snapshot of the data
from the leader replica.</p>
</li>
<li>
@@ -166,6 +166,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -194,8 +196,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/24/multi-master-1-0-0.html
----------------------------------------------------------------------
diff --git a/2016/06/24/multi-master-1-0-0.html b/2016/06/24/multi-master-1-0-0.html
index 294eec2..3891256 100644
--- a/2016/06/24/multi-master-1-0-0.html
+++ b/2016/06/24/multi-master-1-0-0.html
@@ -144,9 +144,9 @@ can be safely enabled in production clusters.</p>
<p>To use replicated masters, a Kudu operator must deploy some number of Kudu
masters, providing the hostname and port number of each master in the group via
-the <code>--master_address</code> command line option. For example, each master in a
+the <code class="highlighter-rouge">--master_address</code> command line option. For example, each master in a
three-node deployment should be started with
-<code>--master_address=<host1:port1>,<host2:port2><host3:port3></code>. In Raft parlance,
+<code class="highlighter-rouge">--master_address=<host1:port1>,<host2:port2><host3:port3></code>. In Raft parlance,
this group of masters is known as a <em>Raft configuration</em>.</p>
<p>At startup, a Raft configuration of masters will hold a leader election and
@@ -191,7 +191,7 @@ clients are also configured with the locations of all masters. Unlike tablet
servers, they always communicate with the leader master as follower masters will
reject client requests. To do this, clients must determine which master is the
leader before sending the first request as well as whenever any request fails
-with a <code>NOT_THE_LEADER</code> error.</p>
+with a <code class="highlighter-rouge">NOT_THE_LEADER</code> error.</p>
<h1 id="remaining-work-for-kudu-10">Remaining work for Kudu 1.0</h1>
@@ -228,6 +228,8 @@ nothing has been implemented yet. Stay tuned!</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -256,8 +258,6 @@ nothing has been implemented yet. Stay tuned!</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/06/27/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/06/27/weekly-update.html b/2016/06/27/weekly-update.html
index f273fae..6f7b17c 100644
--- a/2016/06/27/weekly-update.html
+++ b/2016/06/27/weekly-update.html
@@ -230,6 +230,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -258,8 +260,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/07/01/apache-kudu-0-9-1-released.html
----------------------------------------------------------------------
diff --git a/2016/07/01/apache-kudu-0-9-1-released.html b/2016/07/01/apache-kudu-0-9-1-released.html
index 440e828..101148a 100644
--- a/2016/07/01/apache-kudu-0-9-1-released.html
+++ b/2016/07/01/apache-kudu-0-9-1-released.html
@@ -138,6 +138,8 @@ of 0.9.0 are encouraged to update to the new version at their earliest convenien
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -166,8 +168,6 @@ of 0.9.0 are encouraged to update to the new version at their earliest convenien
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/07/11/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/07/11/weekly-update.html b/2016/07/11/weekly-update.html
index 953a0e8..09e991e 100644
--- a/2016/07/11/weekly-update.html
+++ b/2016/07/11/weekly-update.html
@@ -196,6 +196,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -224,8 +226,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/07/18/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/07/18/weekly-update.html b/2016/07/18/weekly-update.html
index 62c2d8a..217519b 100644
--- a/2016/07/18/weekly-update.html
+++ b/2016/07/18/weekly-update.html
@@ -188,6 +188,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -216,8 +218,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/07/25/asf-graduation.html
----------------------------------------------------------------------
diff --git a/2016/07/25/asf-graduation.html b/2016/07/25/asf-graduation.html
index 37706d4..e73d7c3 100644
--- a/2016/07/25/asf-graduation.html
+++ b/2016/07/25/asf-graduation.html
@@ -164,6 +164,8 @@ Established in 1999, the all-volunteer Foundation oversees more than 350 leading
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -192,8 +194,6 @@ Established in 1999, the all-volunteer Foundation oversees more than 350 leading
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/07/26/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/07/26/weekly-update.html b/2016/07/26/weekly-update.html
index 2486462..2194cb1 100644
--- a/2016/07/26/weekly-update.html
+++ b/2016/07/26/weekly-update.html
@@ -136,8 +136,8 @@ new name and status.</li>
<ul>
<li>
- <p>Dan Burkert contributed a few patches that repackage the Java client under <code>org.apache.kudu</code>
-in place of <code>org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
+ <p>Dan Burkert contributed a few patches that repackage the Java client under <code class="highlighter-rouge">org.apache.kudu</code>
+in place of <code class="highlighter-rouge">org.kududb</code>. This was done in a <strong>backward-incompatible</strong> way, meaning that import
statements will have to be modified in existing Java code to compile against a newer Kudu JAR
version (from 0.10.0 onward). This stems from <a href="http://mail-archives.apache.org/mod_mbox/kudu-dev/201605.mbox/%3CCAGpTDNcJohQBgjzXafXJQdqmBB4sL495p5V_BJRXk_nAGWbzhA@mail.gmail.com%3E">a discussion</a>
initiated in May. It won’t have an impact on C++ or Python users, and it isn’t affecting wire
@@ -147,12 +147,12 @@ compatibility.</p>
<p>Still on the Java-side, J-D Cryans pushed <a href="https://gerrit.cloudera.org/#/c/3055/">a patch</a>
that completely changes how Exceptions are managed. Before this change, users had to introspect
generic Exception objects, making it a guessing game and discouraging good error handling.
-Now, the synchronous client’s methods throw <code>KuduException</code> which packages a <code>Status</code> object
+Now, the synchronous client’s methods throw <code class="highlighter-rouge">KuduException</code> which packages a <code class="highlighter-rouge">Status</code> object
that can be interrogated. This is very similar to how the C++ API works.</p>
<p>Existing code that uses the new Kudu JAR should still compile since this change replaces generic
-<code>Exception</code> with a more specific <code>KuduException</code>. Error handling done by string-matching the
-exception messages should now use the provided <code>Status</code> object.</p>
+<code class="highlighter-rouge">Exception</code> with a more specific <code class="highlighter-rouge">KuduException</code>. Error handling done by string-matching the
+exception messages should now use the provided <code class="highlighter-rouge">Status</code> object.</p>
</li>
<li>
<p>Alexey Serbin’s <a href="https://gerrit.cloudera.org/#/c/3619/">patch</a> that adds Doxygen-based
@@ -160,11 +160,11 @@ documentation was pushed and the new API documentation for C++ developers will b
with the next release.</p>
</li>
<li>
- <p>Todd has made many improvements to the <code>ksck</code> tool over the last week. Building upon Will
-Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code>ksck</code> can
+ <p>Todd has made many improvements to the <code class="highlighter-rouge">ksck</code> tool over the last week. Building upon Will
+Berkeley’s <a href="https://gerrit.cloudera.org/#/c/3632/">WIP patch for KUDU-1516</a>, <code class="highlighter-rouge">ksck</code> can
now detect more problematic situations like if a tablet doesn’t have a majority of replicas on
live tablet servers, or if those replicas aren’t in a good state.
-<code>ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
+<code class="highlighter-rouge">ksck</code> is also <a href="https://gerrit.cloudera.org/#/c/3705/">now faster</a> when run against a large
cluster with a lot of tablets, among other improvements.</p>
</li>
<li>
@@ -196,6 +196,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -224,8 +226,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/08/08/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/08/08/weekly-update.html b/2016/08/08/weekly-update.html
index 0276c74..807d0c2 100644
--- a/2016/08/08/weekly-update.html
+++ b/2016/08/08/weekly-update.html
@@ -195,6 +195,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -223,8 +225,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/08/16/weekly-update.html
----------------------------------------------------------------------
diff --git a/2016/08/16/weekly-update.html b/2016/08/16/weekly-update.html
index e8bfd0e..f3a0523 100644
--- a/2016/08/16/weekly-update.html
+++ b/2016/08/16/weekly-update.html
@@ -168,20 +168,20 @@ from a single-master configuration to a multi-master one. Along the way, he
started building some common infrastructure for command-line tooling.</p>
<p>Since Kudu’s initial release, it has included separate binaries for different
-administrative or operational tools (e.g. <code>kudu-ts-cli</code>, <code>kudu-ksck</code>, <code>kudu-fs_dump</code>,
-<code>log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
+administrative or operational tools (e.g. <code class="highlighter-rouge">kudu-ts-cli</code>, <code class="highlighter-rouge">kudu-ksck</code>, <code class="highlighter-rouge">kudu-fs_dump</code>,
+<code class="highlighter-rouge">log-dump</code>, etc). Despite having similar usage, these tools don’t share much code,
and the separate statically linked binaries make the Kudu packages take more disk
space than strictly necessary.</p>
- <p>Adar’s work has introduced a new top-level <code>kudu</code> binary which exposes a set of subcommands,
-much like the <code>git</code> and <code>docker</code> binaries with which readers may be familiar.
+ <p>Adar’s work has introduced a new top-level <code class="highlighter-rouge">kudu</code> binary which exposes a set of subcommands,
+much like the <code class="highlighter-rouge">git</code> and <code class="highlighter-rouge">docker</code> binaries with which readers may be familiar.
For example, a new tool he has built for dumping peer identifiers from a tablet’s
-consensus metadata is triggered using <code>kudu tablet cmeta print_replica_uuids</code>.</p>
+consensus metadata is triggered using <code class="highlighter-rouge">kudu tablet cmeta print_replica_uuids</code>.</p>
<p>This new tool will be available in the upcoming 0.10.0 release; however, migration
of the existing tools to the new infrastructure has not yet been completed. We
expect that by Kudu 1.0, the old tools will be removed in favor of more subcommands
-of the <code>kudu</code> tool.</p>
+of the <code class="highlighter-rouge">kudu</code> tool.</p>
</li>
<li>
<p>Todd Lipcon picked up the work started by David Alves in July to provide
@@ -205,7 +205,7 @@ was <a href="https://gerrit.cloudera.org/#/c/3840/">committed</a> this week. The
docs will be published as part of the 0.10.0 release.</p>
</li>
<li>
- <p>Alexey also continued work on implementing the <code>AUTO_FLUSH_BACKGROUND</code> write
+ <p>Alexey also continued work on implementing the <code class="highlighter-rouge">AUTO_FLUSH_BACKGROUND</code> write
mode for the C++ client. This feature makes it easier to implement high-throughput
ingest using the C++ API by automatically handling the batching and flushing of writes
based on a configurable buffer size.</p>
@@ -245,6 +245,8 @@ a future post.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -273,8 +275,6 @@ a future post.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/08/23/apache-kudu-0-10-0-released.html
----------------------------------------------------------------------
diff --git a/2016/08/23/apache-kudu-0-10-0-released.html b/2016/08/23/apache-kudu-0-10-0-released.html
index 8bcd629..7cfa6be 100644
--- a/2016/08/23/apache-kudu-0-10-0-released.html
+++ b/2016/08/23/apache-kudu-0-10-0-released.html
@@ -161,6 +161,8 @@ the release notes below.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -189,8 +191,6 @@ the release notes below.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/12782cec/2016/08/23/new-range-partitioning-features.html
----------------------------------------------------------------------
diff --git a/2016/08/23/new-range-partitioning-features.html b/2016/08/23/new-range-partitioning-features.html
index 6e7c3cb..cb0065a 100644
--- a/2016/08/23/new-range-partitioning-features.html
+++ b/2016/08/23/new-range-partitioning-features.html
@@ -221,6 +221,8 @@ be updated to 0.10.</p>
<h3>Recent posts</h3>
<ul>
+ <li> <a href="/2018/09/26/index-skip-scan-optimization-in-kudu.html">Index Skip Scan Optimization in Kudu</a> </li>
+
<li> <a href="/2018/09/11/simplified-pipelines-with-kudu.html">Simplified Data Pipelines with Kudu</a> </li>
<li> <a href="/2018/08/06/getting-started-with-kudu-an-oreilly-title.html">Getting Started with Kudu - an O'Reilly Title</a> </li>
@@ -249,8 +251,6 @@ be updated to 0.10.</p>
<li> <a href="/2016/11/01/weekly-update.html">Apache Kudu Weekly Update November 1st, 2016</a> </li>
- <li> <a href="/2016/10/20/weekly-update.html">Apache Kudu Weekly Update October 20th, 2016</a> </li>
-
</ul>
</div>
</div>