You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2015/07/06 15:12:35 UTC
flink-web git commit: Add new features page
Repository: flink-web
Updated Branches:
refs/heads/asf-site aa0e01afd -> 5dcbfe106
Add new features page
Project: http://git-wip-us.apache.org/repos/asf/flink-web/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink-web/commit/5dcbfe10
Tree: http://git-wip-us.apache.org/repos/asf/flink-web/tree/5dcbfe10
Diff: http://git-wip-us.apache.org/repos/asf/flink-web/diff/5dcbfe10
Branch: refs/heads/asf-site
Commit: 5dcbfe106c6f742bdd5639f5780a2f2ff6f922c0
Parents: aa0e01a
Author: Stephan Ewen <se...@apache.org>
Authored: Mon Jul 6 15:09:39 2015 +0200
Committer: Stephan Ewen <se...@apache.org>
Committed: Mon Jul 6 15:12:02 2015 +0200
----------------------------------------------------------------------
_layouts/features.html | 10 +
css/flink.css | 4 +
features_new.md | 299 ++++++++++++++++++++++++++++
img/features/continuous_streams.png | Bin 0 -> 37006 bytes
img/features/distributed_snapshots.png | Bin 0 -> 32838 bytes
img/features/ecosystem_logos.png | Bin 0 -> 234956 bytes
img/features/exactly_once_state.png | Bin 0 -> 10381 bytes
img/features/iterations.png | Bin 0 -> 135851 bytes
img/features/memory_heap_division.png | Bin 0 -> 28333 bytes
img/features/one_runtime.png | Bin 0 -> 16913 bytes
img/features/optimizer_choice.png | Bin 0 -> 13317 bytes
img/features/stack.png | Bin 0 -> 87387 bytes
img/features/streaming_performance.png | Bin 0 -> 16405 bytes
13 files changed, 313 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/_layouts/features.html
----------------------------------------------------------------------
diff --git a/_layouts/features.html b/_layouts/features.html
new file mode 100644
index 0000000..79178d0
--- /dev/null
+++ b/_layouts/features.html
@@ -0,0 +1,10 @@
+---
+layout: base
+---
+<div class="row">
+ <div class="col-sm-10 col-sm-offset-1">
+
+{{ content }}
+
+ </div>
+</div>
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/css/flink.css
----------------------------------------------------------------------
diff --git a/css/flink.css b/css/flink.css
index 7ad2694..7e04d60 100644
--- a/css/flink.css
+++ b/css/flink.css
@@ -148,6 +148,10 @@ code {
width: 100%;
}
+.img-column {
+ text-align: center;
+}
+
#disqus_thread {
padding-top: 3em;
}
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/features_new.md
----------------------------------------------------------------------
diff --git a/features_new.md b/features_new.md
new file mode 100644
index 0000000..4f49d01
--- /dev/null
+++ b/features_new.md
@@ -0,0 +1,299 @@
+---
+title: "Features"
+layout: features
+---
+
+
+<!-- --------------------------------------------- -->
+<!-- Streaming
+<!-- --------------------------------------------- -->
+
+----
+
+<div class="row" style="padding: 0 0 0 0">
+ <div class="col-sm-12" style="text-align: center;">
+ <h1><b>Streaming</b></h1>
+ </div>
+</div>
+
+----
+
+<!-- High Performance -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="performance"><i>High Performance</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink's data streaming runtime achieves high throughput rates and low latencies with little configuration.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/streaming_performance.png" alt="Performance of data streaming applications" style="width:75%" />
+ </div>
+</div>
+
+----
+
+<!-- Exactly-once Semantics -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="exactly_once"><i>Exactly-once Semantics for Stateful Computations</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Streaming applications can maintain custom state during their computation.</p>
+ <p class="lead">Flink's checkpointing mechanism ensures <i>exactly once</i> semantics for the state in the presence of failures.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/exactly_once_state.png" alt="Exactly-once Semantics for Stateful Computations" style="width:40%" />
+ </div>
+</div>
+
+----
+
+<!-- Continuous streaming -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="streaming_model"><i>Continuous Streaming Model with Flow Control</i></h1>
+ </div>
+</div>
+
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Data streaming applications are executed with continuous (long lived) operators.</p>
+ <p class="lead">Flink's streaming runtime has natural flow control: Slow downstream operators backpressure faster upstream operators.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/continuous_streams.png" alt="Continuous Streaming Model" style="width:60%" />
+ </div>
+</div>
+
+----
+
+<!-- Lightweight distributed snapshots -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="snapshots"><i>Fault-tolerance via Lightweight Distributed Snapshots</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink's fault tolerance mechanism is based on Chandy-Lamport distributed snapshots.</p>
+ <p class="lead">The mechanism is lightweight, allowing the system to maintain high throughput rates and provide strong consistency guarantees at the same time.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/distributed_snapshots.png" alt="Lightweight Distributed Snapshots" style="width:40%" />
+ </div>
+</div>
+
+----
+
+<!-- --------------------------------------------- -->
+<!-- Batch
+<!-- --------------------------------------------- -->
+
+<div class="row" style="padding: 0 0 0 0">
+ <div class="col-sm-12" style="text-align: center;">
+ <h1><b>Batch and Streaming in One System</b></h1>
+ </div>
+</div>
+
+----
+
+<!-- One Runtime for Streaming and Batch Processing -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="one_runtime"><i>One Runtime for Streaming and Batch Processing</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink uses one common runtime for data streaming applications and batch processing applications.</p>
+ <p class="lead">Batch processing applications run efficiently as special cases of stream processing applications.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/one_runtime.png" alt="Unified Runtime for Batch and Stream Data Analysis" style="width:75%" />
+ </div>
+</div>
+
+----
+
+
+<!-- Memory Management -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="memory_management"><i>Memory Management</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink implements its own memory management inside the JVM.</p>
+ <p class="lead">Applications scale to data sizes beyond main memory and experience less garbage collection overhead.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/memory_heap_division.png" alt="Managed JVM Heap" style="width:50%" />
+ </div>
+</div>
+
+----
+
+<!-- Iterations -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="iterations"><i>Iterations and Delta Iterations</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink has dedicated support for iterative computations (as in machine learning and graph analysis).</p>
+ <p class="lead">Delta iterations can exploit computational dependencies for faster convergence.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/iterations.png" alt="Performance of iterations and delta iterations" style="width:75%" />
+ </div>
+</div>
+
+----
+
+<!-- Optimizer -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="optimizer"><i>Program Optimizer</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Batch programs are automatically optimized to exploit situations where expensive operations (like shuffles and sorts) can be avoided, and when intermediate data should be cached.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/optimizer_choice.png" alt="Optimizer choosing between different execution strategies" style="width:40%" />
+ </div>
+</div>
+
+----
+
+<!-- --------------------------------------------- -->
+<!-- APIs and Libraries
+<!-- --------------------------------------------- -->
+
+<div class="row" style="padding: 0 0 0 0">
+ <div class="col-sm-12" style="text-align: center;">
+ <h1><b>APIs and Libraries</b></h1>
+ </div>
+</div>
+
+----
+
+<!-- Batch Processing API -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="batch_api"><i>Batch Processing Applications</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-5">
+ <p class="lead">Flink's <i>DataSet</i> API lets you write beautiful type-safe and maintainable code in Java or Scala. It supports a wide range of data types beyond key/value pairs, and a wealth of operators.</p>
+ <p class="lead">The example shows the core loop of the PageRank algorithm for graphs.</p>
+ </div>
+ <div class="col-sm-7">
+{% highlight scala %}
+case class Page(pageId: Long, rank: Double)
+case class Adjacency(id: Long, neighbors: Array[Long])
+
+val result = initialRanks.iterate(30) { pages =>
+ pages.join(adjacency).where("pageId").equalTo("pageId") {
+
+ (page, adj, out : Collector[Page]) => {
+ out.collect(Page(page.id, 0.15 / numPages))
+
+ for (n <- adj.neighbors) {
+ out.collect(Page(n, 0.85*page.rank/adj.neighbors.length))
+ }
+ }
+ }
+ .groupBy("pageId").sum("rank")
+}
+{% endhighlight %}
+ </div>
+</div>
+
+----
+
+<!-- Data Streaming API -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="streaming_api"><i>Streaming Data Applications</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-5">
+ <p class="lead">The <i>DataStream</i> API supports functional transformations on data streams, with user-defined state, and flexible windows.</p>
+ <p class="lead">The example shows how to compute a sliding historam of word occurrences of a data stream of texts.</p>
+ </div>
+ <div class="col-sm-7">
+ <p class="lead">WindowWordCount in Flink's DataStream API</p>
+{% highlight scala %}
+case class Word(word: String, freq: Long)
+
+val texts: DataStream[String] = ...
+
+val counts = text
+ .flatMap { line => line.split("\\W+") }
+ .map { token => Word(token, 1) }
+ .groupBy("word")
+ .window(Time.of(5, SECONDS)).every(Time.of(1, SECONDS))
+ .sum("freq")
+{% endhighlight %}
+ </div>
+</div>
+
+----
+
+<!-- Library Ecosystem -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="libraries"><i>Library Ecosystem</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink's stack offers libraries with high-level APIs for different use cases: Machine Learning, Graph Analytics, and Relational Data Processing.</p>
+ <p class="lead">The libraries are currently in <i>beta</i> status and are heavily developed.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/stack.png" alt="Flink Stack with Libraries" style="width:60%" />
+ </div>
+</div>
+
+----
+
+<!-- --------------------------------------------- -->
+<!-- Ecosystem
+<!-- --------------------------------------------- -->
+
+<div class="row" style="padding: 0 0 0 0">
+ <div class="col-sm-12" style="text-align: center;">
+ <h1><b>Ecosystem</b></h1>
+ </div>
+</div>
+
+----
+
+<!-- Ecosystem -->
+<div class="row" style="padding: 0 0 2em 0">
+ <div class="col-sm-12">
+ <h1 id="ecosystem"><i>Broad Integration</i></h1>
+ </div>
+</div>
+<div class="row">
+ <div class="col-sm-6">
+ <p class="lead">Flink is integrated with many other projects in the open-source data processing ecosystem.</p>
+ <p class="lead">Flink runs on YARN, works with HDFS, streams data from Kafka, can execute Hadoop program code, and connects to various other data storage systems.</p>
+ </div>
+ <div class="col-sm-6 img-column">
+ <img src="{{ site.baseurl }}/img/features/ecosystem_logos.png" alt="Other projects that Flink is integrated with" style="width:75%" />
+ </div>
+</div>
+
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/continuous_streams.png
----------------------------------------------------------------------
diff --git a/img/features/continuous_streams.png b/img/features/continuous_streams.png
new file mode 100644
index 0000000..a4bb16b
Binary files /dev/null and b/img/features/continuous_streams.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/distributed_snapshots.png
----------------------------------------------------------------------
diff --git a/img/features/distributed_snapshots.png b/img/features/distributed_snapshots.png
new file mode 100644
index 0000000..9594f27
Binary files /dev/null and b/img/features/distributed_snapshots.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/ecosystem_logos.png
----------------------------------------------------------------------
diff --git a/img/features/ecosystem_logos.png b/img/features/ecosystem_logos.png
new file mode 100644
index 0000000..ad36d8b
Binary files /dev/null and b/img/features/ecosystem_logos.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/exactly_once_state.png
----------------------------------------------------------------------
diff --git a/img/features/exactly_once_state.png b/img/features/exactly_once_state.png
new file mode 100644
index 0000000..6b6d4df
Binary files /dev/null and b/img/features/exactly_once_state.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/iterations.png
----------------------------------------------------------------------
diff --git a/img/features/iterations.png b/img/features/iterations.png
new file mode 100644
index 0000000..989564e
Binary files /dev/null and b/img/features/iterations.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/memory_heap_division.png
----------------------------------------------------------------------
diff --git a/img/features/memory_heap_division.png b/img/features/memory_heap_division.png
new file mode 100644
index 0000000..e0807b0
Binary files /dev/null and b/img/features/memory_heap_division.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/one_runtime.png
----------------------------------------------------------------------
diff --git a/img/features/one_runtime.png b/img/features/one_runtime.png
new file mode 100644
index 0000000..94453a6
Binary files /dev/null and b/img/features/one_runtime.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/optimizer_choice.png
----------------------------------------------------------------------
diff --git a/img/features/optimizer_choice.png b/img/features/optimizer_choice.png
new file mode 100644
index 0000000..e64ca1e
Binary files /dev/null and b/img/features/optimizer_choice.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/stack.png
----------------------------------------------------------------------
diff --git a/img/features/stack.png b/img/features/stack.png
new file mode 100644
index 0000000..2c34722
Binary files /dev/null and b/img/features/stack.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/5dcbfe10/img/features/streaming_performance.png
----------------------------------------------------------------------
diff --git a/img/features/streaming_performance.png b/img/features/streaming_performance.png
new file mode 100644
index 0000000..c6840f3
Binary files /dev/null and b/img/features/streaming_performance.png differ