You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2018/06/21 15:29:06 UTC
[07/12] flink-web git commit: Rebuild website
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/flink-applications.html
----------------------------------------------------------------------
diff --git a/content/flink-applications.html b/content/flink-applications.html
new file mode 100644
index 0000000..70525bf
--- /dev/null
+++ b/content/flink-applications.html
@@ -0,0 +1,385 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: What is Apache Flink?</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+ <!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li class="active"><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Quickstart -->
+ <li>
+ <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">1.5 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.6 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/how-to-contribute.html">How to Contribute</a></li>
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ </ul>
+
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <h1>What is Apache Flink?</h1>
+
+ <hr />
+
+<div class="row">
+ <div class="col-sm-12" style="background-color: #f8f8f8;">
+ <h2>
+ <a href="/flink-architecture.html">Architecture</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ Applications
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-operations.html">Operations</a>
+ </h2>
+ </div>
+</div>
+<hr />
+
+<p>Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Flink provides multiple APIs at different levels of abstraction and offers dedicated libraries for common use cases.</p>
+
+<p>Here, we present Flink’s easy-to-use and expressive APIs and libraries.</p>
+
+<h2 id="building-blocks-for-streaming-applications">Building Blocks for Streaming Applications</h2>
+
+<p>The types of applications that can be built with and executed by a stream processing framework are defined by how well the framework controls <em>streams</em>, <em>state</em>, and <em>time</em>. In the following, we describe these building blocks for stream processing applications and explain Flink’s approaches to handle them.</p>
+
+<h3 id="streams">Streams</h3>
+
+<p>Obviously, streams are a fundamental aspect of stream processing. However, streams can have different characteristics that affect how a stream can and should be processed. Flink is a versatile processing framework that can handle any kind of stream.</p>
+
+<ul>
+ <li><strong>Bounded</strong> and <strong>unbounded</strong> streams: Streams can be unbounded or bounded, i.e., fixed-sized data sets. Flink has sophisticated features to process unbounded streams, but also dedicated operators to efficiently process bounded streams.</li>
+ <li><strong>Real-time</strong> and <strong>recorded</strong> streams: All data are generated as streams. There are two ways to process the data. Processing it in real-time as it is generated or persisting the stream to a storage system, e.g., a file system or object store, and processed it later. Flink applications can process recorded or real-time streams.</li>
+</ul>
+
+<h3 id="state">State</h3>
+
+<p>Every non-trivial streaming application is stateful, i.e., only applications that apply transformations on individual events do not require state. Any application that runs basic business logic needs to remember events or intermediate results to access them at a later point in time, for example when the next event is received or after a specific time duration.</p>
+
+<div class="row front-graphic">
+ <img src="/img/function-state.png" width="350px" />
+</div>
+
+<p>Application state is a first-class citizen in Flink. You can see that by looking at all the features that Flink provides in the context of state handling.</p>
+
+<ul>
+ <li><strong>Multiple State Primitives</strong>: Flink provides state primitives for different data structures, such as atomic values, lists, or maps. Developers can choose the state primitive that is most efficient based on the access pattern of the function.</li>
+ <li><strong>Pluggable State Backends</strong>: Application state is managed in and checkpointed by a pluggable state backend. Flink features different state backends that store state in memory or in <a href="https://rocksdb.org/">RocksDB</a>, an efficient embedded on-disk data store. Custom state backends can be plugged in as well.</li>
+ <li><strong>Exactly-once state consistency</strong>: Flink’s checkpointing and recovery algorithms guarantee the consistency of application state in case of a failure. Hence, failures are transparently handled and do not affect the correctness of an application.</li>
+ <li><strong>Very Large State</strong>: Flink is able to maintain application state of several terabytes in size due to its asynchronous and incremental checkpoint algorithm.</li>
+ <li><strong>Scalable Applications</strong>: Flink supports scaling of stateful applications by redistributing the state to more or fewer workers.</li>
+</ul>
+
+<h3 id="time">Time</h3>
+
+<p>Time is another important ingredient of streaming applications. Most event streams have inherent time semantics because each event is produced at a specific point in time. Moreover, many common stream computations are based on time, such as windows aggregations, sessionization, pattern detection, and time-based joins. An important aspect of stream processing is how an application measures time, i.e., the difference of event-time and processing-time.</p>
+
+<p>Flink provides a rich set of time-related features.</p>
+
+<ul>
+ <li><strong>Event-time Mode</strong>: Applications that process streams with event-time semantics compute results based on timestamps of the events. Thereby, event-time processing allows for accurate and consistent results regardless whether recorded or real-time events are processed.</li>
+ <li><strong>Watermark Support</strong>: Flink employs watermarks to reason about time in event-time applications. Watermarks are also a flexible mechanism to trade-off the latency and completeness of results.</li>
+ <li><strong>Late Data Handling</strong>: When processing streams in event-time mode with watermarks, it can happen that a computation has been completed before all associated events have arrived. Such events are called late events. Flink features multiple options to handle late events, such as rerouting them via side outputs and updating previously completed results.</li>
+ <li><strong>Processing-time Mode</strong>: In addition to its event-time mode, Flink also supports processing-time semantics which performs computations as triggered by the wall-clock time of the processing machine. The processing-time mode can be suitable for certain applications with strict low-latency requirements that can tolerate approximate results.</li>
+</ul>
+
+<h2 id="layered-apis">Layered APIs</h2>
+
+<p>Flink provides three layered APIs. Each API offers a different trade-off between conciseness and expressiveness and targets different use cases.</p>
+
+<div class="row front-graphic">
+ <img src="/img/api-stack.png" width="500px" />
+</div>
+
+<p>We briefly present each API, discuss its applications, and show a code example.</p>
+
+<h3 id="the-processfunctions">The ProcessFunctions</h3>
+
+<p><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html">ProcessFunctions</a> are the most expressive function interfaces that Flink offers. Flink provides ProcessFunctions to process individual events from one or two input streams or events that were grouped in a window. ProcessFunctions provide fine-grained control over time and state. A ProcessFunction can arbitrarily modify its state and register timers that will trigger a callback function in the future. Hence, ProcessFunctions can implement complex per-event business logic as required for many <a href="/usecases.html#eventDrivenApps">stateful event-driven applications</a>.</p>
+
+<p>The following example shows a <code>KeyedProcessFunction</code> that operates on a <code>KeyedStream</code> and matches <code>START</code> and <code>END</code> events. When a <code>START</code> event is received, the function remembers its timestamp in state and registers a timer in four hours. If an <code>END</code> event is received before the timer fires, the function computes the duration between <code>END</code> and <code>START</code> event, clears the state, and returns the value. Otherwise, the timer just fires and clears the state.</p>
+
+<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="cm">/**</span>
+<span class="cm"> * Matches keyed START and END events and computes the difference between </span>
+<span class="cm"> * both elements' timestamps. The first String field is the key attribute, </span>
+<span class="cm"> * the second String attribute marks START and END events.</span>
+<span class="cm"> */</span>
+<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">StartEndDuration</span>
+ <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">>,</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="o">{</span>
+
+ <span class="kd">private</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">Long</span><span class="o">></span> <span class="n">startTime</span><span class="o">;</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// obtain state handle</span>
+ <span class="n">startTime</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">getState</span><span class="o">(</span><span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o"><</span><span class="n">Long</span><span class="o">>(</span><span class="s">"startTime"</span><span class="o">,</span> <span class="n">Long</span><span class="o">.</span><span class="na">class</span><span class="o">));</span>
+ <span class="o">}</span>
+
+ <span class="cm">/** Called for each processed event. */</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">></span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+
+ <span class="k">switch</span> <span class="o">(</span><span class="n">in</span><span class="o">.</span><span class="na">f1</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">case</span> <span class="s">"START"</span><span class="o">:</span>
+ <span class="c1">// set the start time if we receive a start event.</span>
+ <span class="n">startTime</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">ctx</span><span class="o">.</span><span class="na">timestamp</span><span class="o">());</span>
+ <span class="c1">// register a timer in four hours from the start event.</span>
+ <span class="n">ctx</span><span class="o">.</span><span class="na">timerService</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">registerEventTimeTimer</span><span class="o">(</span><span class="n">ctx</span><span class="o">.</span><span class="na">timestamp</span><span class="o">()</span> <span class="o">+</span> <span class="mi">4</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">1000</span><span class="o">);</span>
+ <span class="k">break</span><span class="o">;</span>
+ <span class="k">case</span> <span class="s">"END"</span><span class="o">:</span>
+ <span class="c1">// emit the duration between start and end event</span>
+ <span class="n">Long</span> <span class="n">sTime</span> <span class="o">=</span> <span class="n">startTime</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">sTime</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">in</span><span class="o">.</span><span class="na">f0</span><span class="o">,</span> <span class="n">ctx</span><span class="o">.</span><span class="na">timestamp</span><span class="o">()</span> <span class="o">-</span> <span class="n">sTime</span><span class="o">));</span>
+ <span class="c1">// clear the state</span>
+ <span class="n">startTime</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span>
+ <span class="o">}</span>
+ <span class="k">default</span><span class="o">:</span>
+ <span class="c1">// do nothing</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="cm">/** Called when a timer fires. */</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span>
+ <span class="kt">long</span> <span class="n">timestamp</span><span class="o">,</span>
+ <span class="n">OnTimerContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span>
+
+ <span class="c1">// Timeout interval exceeded. Cleaning up the state.</span>
+ <span class="n">startTime</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The example illustrates the expressive power of the <code>KeyedProcessFunction</code> but also highlights that it is a rather verbose interface.</p>
+
+<h3 id="the-datastream-api">The DataStream API</h3>
+
+<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_api.html">DataStream API</a> provides primitives for many common stream processing operations, such as windowing, record-at-a-time transformations, and enriching events by querying an external data store. The DataStream API is available for Java and Scala and is based on functions, such as <code>map()</code>, <code>reduce()</code>, and <code>aggregate()</code>. Functions can be defined by extending interfaces or as Java or Scala lambda functions.</p>
+
+<p>The following example shows how to sessionize a clickstream and count the number of clicks per session.</p>
+
+<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// a stream of website clicks</span>
+<span class="n">DataStream</span><span class="o"><</span><span class="n">Click</span><span class="o">></span> <span class="n">clicks</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="n">DataStream</span><span class="o"><</span><span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>></span> <span class="n">result</span> <span class="o">=</span> <span class="n">clicks</span>
+ <span class="c1">// project clicks to userId and add a 1 for counting</span>
+ <span class="o">.</span><span class="na">map</span><span class="o">(</span>
+ <span class="c1">// define function by implementing the MapFunction interface.</span>
+ <span class="k">new</span> <span class="n">MapFunction</span><span class="o"><</span><span class="n">Click</span><span class="o">,</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">>>()</span> <span class="o">{</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">Tuple2</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">></span> <span class="nf">map</span><span class="o">(</span><span class="n">Click</span> <span class="n">click</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">click</span><span class="o">.</span><span class="na">userId</span><span class="o">,</span> <span class="mi">1L</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">})</span>
+ <span class="c1">// key by userId (field 0)</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
+ <span class="c1">// define session window with 30 minute gap</span>
+ <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="n">EventTimeSessionWindows</span><span class="o">.</span><span class="na">withGap</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">minutes</span><span class="o">(</span><span class="mi">30L</span><span class="o">)))</span>
+ <span class="c1">// count clicks per session. Define function as lambda function.</span>
+ <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="o">-></span> <span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">a</span><span class="o">.</span><span class="na">f0</span><span class="o">,</span> <span class="n">a</span><span class="o">.</span><span class="na">f1</span> <span class="o">+</span> <span class="n">b</span><span class="o">.</span><span class="na">f1</span><span class="o">));</span></code></pre></div>
+
+<h3 id="sql-amp-table-api">SQL & Table API</h3>
+
+<p>Flink features two relational APIs, the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/index.html">Table API and SQL</a>. Both APIs are unified APIs for batch and stream processing, i.e., queries are executed with the same semantics on unbounded, real-time streams or bounded, recorded streams and produce the same results. The Table API and SQL leverage <a href="https://calcite.apache.org">Apache Calcite</a> for parsing, validation, and query optimization. They can be seamlessly integrated with the DataStream and DataSet APIs and support user-defined scalar, aggregate, and table-valued functions.</p>
+
+<p>Flink’s relational APIs are designed to ease the definition of <a href="/usecases.html#analytics">data analytics</a>, <a href="/usecases.html#pipelines">data pipelining, and ETL applications</a>.</p>
+
+<p>The following example shows the SQL query to sessionize a clickstream and count the number of clicks per session. This is the same use case as in the example of the DataStream API.</p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">SELECT</span> <span class="n">userId</span><span class="p">,</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
+<span class="k">FROM</span> <span class="n">clicks</span>
+<span class="k">GROUP</span> <span class="k">BY</span> <span class="k">SESSION</span><span class="p">(</span><span class="n">clicktime</span><span class="p">,</span> <span class="nb">INTERVAL</span> <span class="s1">'30'</span> <span class="k">MINUTE</span><span class="p">),</span> <span class="n">userId</span></code></pre></div>
+
+<h2 id="libraries">Libraries</h2>
+
+<p>Flink features several libraries for common data processing use cases. The libraries are typically embedded in an API and not fully self-contained. Hence, they can benefit from all features of the API and be integrated with other libraries.</p>
+
+<ul>
+ <li>
+ <p><strong><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/cep.html">Complex Event Processing (CEP)</a></strong>: Pattern detection is a very common use case for event stream processing. Flink’s CEP library provides an API to specify patterns of events (think of regular expressions or state machines). The CEP library is integrated with Flink’s DataStream API, such that patterns are evaluated on DataStreams. Applications for the CEP library include network intrusion detection, business process monitoring, and fraud detection.</p>
+ </li>
+ <li>
+ <p><strong><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/batch/index.html">DataSet API</a></strong>: The DataSet API is Flink’s core API for batch processing applications. The primitives of the DataSet API include <em>map</em>, <em>reduce</em>, <em>(outer) join</em>, <em>co-group</em>, and <em>iterate</em>. All operations are backed by algorithms and data structures that operate on serialized data in memory and spill to disk if the data size exceed the memory budget. The data processing algorithms of Flink’s DataSet API are inspired by traditional database operators, such as hybrid hash-join or external merge-sort.</p>
+ </li>
+ <li>
+ <p><strong><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/gelly/index.html">Gelly</a></strong>: Gelly is a library for scalable graph processing and analysis. Gelly is implemented on top of and integrated with the DataSet API. Hence, it benefits from its scalable and robust operators. Gelly features <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/gelly/library_methods.html">built-in algorithms</a>, such as label propagation, triangle enumeration, and page rank, but provides also a <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/gelly/graph_api.html">Graph API</a> that eases the implementation of custom graph algorithms.</p>
+ </li>
+</ul>
+
+<hr />
+
+<div class="row">
+ <div class="col-sm-12" style="background-color: #f8f8f8;">
+ <h2>
+ <a href="/flink-architecture.html">Architecture</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ Applications
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-operations.html">Operations</a>
+ </h2>
+ </div>
+</div>
+<hr />
+
+
+
+ </div>
+</div>
+
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2017 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/flink-architecture.html
----------------------------------------------------------------------
diff --git a/content/flink-architecture.html b/content/flink-architecture.html
new file mode 100644
index 0000000..b009b97
--- /dev/null
+++ b/content/flink-architecture.html
@@ -0,0 +1,284 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: What is Apache Flink?</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+ <!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li class="active"><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Quickstart -->
+ <li>
+ <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">1.5 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.6 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/how-to-contribute.html">How to Contribute</a></li>
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ </ul>
+
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <h1>What is Apache Flink?</h1>
+
+ <hr />
+
+<div class="row">
+ <div class="col-sm-12" style="background-color: #f8f8f8;">
+ <h2>
+ Architecture
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-applications.html">Applications</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-operations.html">Operations</a>
+ </h2>
+ </div>
+</div>
+<hr />
+
+<p>Apache Flink is a framework and distributed processing engine for stateful computations over <em>unbounded and bounded</em> data streams. Flink has been designed to run in <em>all common cluster environments</em>, perform computations at <em>in-memory speed</em> and at <em>any scale</em>.</p>
+
+<p>Here, we explain important aspects of Flink’s architecture.</p>
+
+<!--
+<div class="row front-graphic">
+ <img src="/img/flink-home-graphic-update3.png" width="800px" />
+</div>
+-->
+
+<h2 id="process-unbounded-and-bounded-data">Process Unbounded and Bounded Data</h2>
+
+<p>Any kind of data is produced as a stream of events. Credit card transactions, sensor measurements, machine logs, or user interactions on a website or mobile application, all of these data are generated as a stream.</p>
+
+<p>Data can be processed as <em>unbounded</em> or <em>bounded</em> streams.</p>
+
+<ol>
+ <li>
+ <p><strong>Unbounded streams</strong> have a start but no defined end. They do not terminate and provide data as it is generated. Unbounded streams must be continuously processed, i.e., events must be promptly handled after they have been ingested. It is not possible to wait for all input data to arrive because the input is unbounded and will not be complete at any point in time. Processing unbounded data often requires that events are ingested in a specific order, such as the order in which events occurred, to be able to reason about result completeness.</p>
+ </li>
+ <li>
+ <p><strong>Bounded streams</strong> have a defined start and end. Bounded streams can be processed by ingesting all data before performing any computations. Ordered ingestion is not required to process bounded streams because a bounded data set can always be sorted. Processing of bounded streams is also known as batch processing.</p>
+ </li>
+</ol>
+
+<div class="row front-graphic">
+ <img src="/img/bounded-unbounded.png" width="600px" />
+</div>
+
+<p><strong>Apache Flink excels at processing unbounded and bounded data sets.</strong> Precise control of time and state enable Flink’s runtime to run any kind of application on unbounded streams. Bounded streams are internally processed by algorithms and data structures that are specifically designed for fixed sized data sets, yielding excellent performance.</p>
+
+<p>Convince yourself by exploring the <a href="/usecases.html">use cases</a> that have been built on top of Flink.</p>
+
+<h2 id="deploy-applications-anywhere">Deploy Applications Anywhere</h2>
+
+<p>Apache Flink is a distributed system and requires compute resources in order to execute applications. Flink integrates with all common cluster resource managers such as <a href="https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html">Hadoop YARN</a>, <a href="https://mesos.apache.org">Apache Mesos</a>, and <a href="https://kubernetes.io/">Kubernetes</a> but can also be setup to run as a stand-alone cluster.</p>
+
+<p>Flink is designed to work well each of the previously listed resource managers. This is achieved by resource-manager-specific deployment modes that allow Flink to interact with each resource manager in its idiomatic way.</p>
+
+<p>When deploying a Flink application, Flink automatically identifies the required resources based on the application’s configured parallelism and requests them from the resource manager. In case of a failure, Flink replaces the failed container by requesting new resources. All communication to submit or control an application happens via REST calls. This eases the integration of Flink in many environments.</p>
+
+<!-- Add this section once library deployment mode is supported. -->
+<!--
+
+Flink features two deployment modes for applications, the *framework mode* and the *library mode*.
+
+* In the **framework deployment mode**, a client submits a Flink application against a running Flink service that takes care of executing the application. This is the common deployment model for most data processing frameworks, query engines, or database systems.
+
+* In the **library deployment mode**, a Flink application is packaged together with the Flink master executables into a (Docker) image. Another job-independent image contains the Flink worker executables. When a container is started from the job image, the Flink master process is started and the embedded application is automatically loaded. Containers started from the worker image, bootstrap Flink worker processes which automatically connect to the master process. A container manager such as Kubernetes monitors the running containers and automatically restarts failed containers. In this mode, you don't have to setup and maintain a Flink service in your cluster. Instead you package Flink as a library with your application. This model is very popular for deploying microservices.
+
+<div class="row front-graphic">
+ <img src="/img/deployment-modes.png" width="600px" />
+</div>
+
+-->
+
+<h2 id="run-applications-at-any-scale">Run Applications at any Scale</h2>
+
+<p>Flink is designed to run stateful streaming applications at any scale. Applications are parallelized into possibly thousands of tasks that are distributed and concurrently executed in a cluster. Therefore, an application can leverage virtually unlimited amounts of CPUs, main memory, disk and network IO. Moreover, Flink easily maintains very large application state. Its asynchronous and incremental checkpointing algorithm ensures minimal impact on processing latencies while guaranteeing exactly-once state consistency.</p>
+
+<p><a href="/poweredby.html">Users reported impressive scalability numbers</a> for Flink applications running in their production environments, such as</p>
+
+<ul>
+ <li>applications processing <strong>multiple trillions of events per day</strong>,</li>
+ <li>applications maintaining <strong>multiple terabytes of state</strong>, and</li>
+ <li>applications <strong>running on thousands of cores</strong>.</li>
+</ul>
+
+<h2 id="leverage-in-memory-performance">Leverage In-Memory Performance</h2>
+
+<p>Stateful Flink applications are optimized for local state access. Task state is always maintained in memory or, if the state size exceeds the available memory, in access-efficient on-disk data structures. Hence, tasks perform all computations by accessing local, often in-memory, state yielding very low processing latencies. Flink guarantees exactly-once state consistency in case of failures by periodically and asynchronously checkpointing the local state to durable storage.</p>
+
+<div class="row front-graphic">
+ <img src="/img/local-state.png" width="600px" />
+</div>
+
+<hr />
+
+<div class="row">
+ <div class="col-sm-12" style="background-color: #f8f8f8;">
+ <h2>
+ Architecture
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-applications.html">Applications</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-operations.html">Operations</a>
+ </h2>
+ </div>
+</div>
+<hr />
+
+
+
+ </div>
+</div>
+
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2017 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/flink-operations.html
----------------------------------------------------------------------
diff --git a/content/flink-operations.html b/content/flink-operations.html
new file mode 100644
index 0000000..cdc8350
--- /dev/null
+++ b/content/flink-operations.html
@@ -0,0 +1,255 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: What is Apache Flink?</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+ <!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li class="active"><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Quickstart -->
+ <li>
+ <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">1.5 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.6 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/how-to-contribute.html">How to Contribute</a></li>
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ </ul>
+
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <h1>What is Apache Flink?</h1>
+
+ <hr />
+
+<div class="row">
+ <div class="col-sm-12" style="background-color: #f8f8f8;">
+ <h2>
+ <a href="/flink-architecture.html">Architecture</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-applications.html">Applications</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ Operations
+ </h2>
+ </div>
+</div>
+<hr />
+
+<p>Apache Flink is a framework for stateful computations over unbounded and bounded data streams. Since many streaming applications are designed to run continuously with minimal downtime, a stream processor must provide excellent failure recovery, as well as, tooling to monitor and maintain applications while they are running.</p>
+
+<p>Apache Flink puts a strong focus on the operational aspects of stream processing. Here, we explain Flink’s failure recovery mechanism and present its features to manage and supervise running applications.</p>
+
+<h2 id="run-your-applications-non-stop-247">Run Your Applications Non-Stop 24/7</h2>
+
+<p>Machine and process failures are ubiquitous in distributed systems. A distributed stream processors like Flink must recover from failures in order to be able to run streaming applications 24/7. Obviously, this does not only mean to restart an application after a failure but also to ensure that its internal state remains consistent, such that the application can continue processing as if the failure had never happened.</p>
+
+<p>Flink provides a several features to ensure that applications keep running and remain consistent:</p>
+
+<ul>
+ <li><strong>Consistent Checkpoints</strong>: Flink’s recovery mechanism is based on consistent checkpoints of an application’s state. In case of a failure, the application is restarted and its state is loaded from the latest checkpoint. In combination with resettable stream sources, this feature can guarantee <em>exactly-once state consistency</em>.</li>
+ <li><strong>Efficient Checkpoints</strong>: Checkpointing the state of an application can be quite expensive if the application maintains terabytes of state. Flink’s can perform asynchronous and incremental checkpoints, in order to keep the impact of checkpoints on the application’s latency SLAs very small.</li>
+ <li><strong>End-to-End Exactly-Once</strong>: Flink features transactional sinks for specific storage systems that guarantee that data is only written out exactly once, even in case of failures.</li>
+ <li><strong>Integration with Cluster Managers</strong>: Flink is tightly integrated with cluster managers, such as <a href="https://hadoop.apache.org">Hadoop YARN</a>, <a href="https://mesos.apache.org">Mesos</a>, or <a href="https://kubernetes.io">Kubernetes</a>. When a process fails, a new process is automatically started to take over its work.</li>
+ <li><strong>High-Availability Setup</strong>: Flink feature a high-availability mode that eliminates all single-points-of-failure. The HA-mode is based on <a href="https://zookeeper.apache.org">Apache ZooKeeper</a>, a battle-proven service for reliable distributed coordination.</li>
+</ul>
+
+<h2 id="update-migrate-suspend-and-resume-your-applications">Update, Migrate, Suspend, and Resume Your Applications</h2>
+
+<p>Streaming applications that power business-critical services need to be maintained. Bugs need to be fixed and improvements or new features need to be implemented. However, updating a stateful streaming application is not trivial. Often one cannot simply stop the applications and restart an fixed or improved version because one cannot afford to lose the state of the application.</p>
+
+<p>Flink’s <em>Savepoints</em> are a unique and powerful feature that solves the issue of updating stateful applications and many other related challenges. A savepoint is a consistent snapshot of an application’s state and therefore very similar to a checkpoint. However in contrast to checkpoints, savepoints need to be manually triggered and are not automatically removed when an application is stopped. A savepoint can be used to start a state-compatible application and initialize its state. Savepoints enable the following features:</p>
+
+<ul>
+ <li><strong>Application Evolution</strong>: Savepoints can be used to evolve applications. A fixed or improved version of an application can be restarted from a savepoint that was taken from a previous version of the application. It is also possible to start the application from an earlier point in time (given such a savepoint exists) to repair incorrect results produced by the flawed version.</li>
+ <li><strong>Cluster Migration</strong>: Using savepoints, applications can be migrated (or cloned) to different clusters.</li>
+ <li><strong>Flink Version Updates</strong>: An application can be migrated to run on a new Flink version using a savepoint.</li>
+ <li><strong>Application Scaling</strong>: Savepoints can be used to increase or decrease the parallelism of an application.</li>
+ <li><strong>A/B Tests and What-If Scenarios</strong>: The performance or quality of two (or more) different versions of an application can be compared by starting all versions from the same savepoint.</li>
+ <li><strong>Pause and Resume</strong>: An application can be paused by taking a savepoint and stopping it. At any later point in time, the application can be resumed from the savepoint.</li>
+ <li><strong>Archiving</strong>: Savepoints can be archived to be able to reset the state of an application to an earlier point in time.</li>
+</ul>
+
+<h2 id="monitor-and-control-your-applications">Monitor and Control Your Applications</h2>
+
+<p>Just like any other service, continuously running streaming applications need to be supervised and integrated into the operations infrastructure, i.e., monitoring and logging services, of an organization. Monitoring helps to anticipate problems and react ahead of time. Logging enables root-cause analysis to investigate failures. Finally, easily accessible interfaces to control running applications are an important feature.</p>
+
+<p>Flink integrates nicely with many common logging and monitoring services and provides a REST API to control applications and query information.</p>
+
+<ul>
+ <li><strong>Web UI</strong>: Flink features a web UI to inspect, monitor, and debug running applications. It can also be used to submit executions for execution or cancel them.</li>
+ <li><strong>Logging</strong>: Flink implements the popular slf4j logging interface and integrates with the logging frameworks <a href="https://logging.apache.org/log4j/2.x/">log4j</a> or <a href="https://logback.qos.ch/">logback</a>.</li>
+ <li><strong>Metrics</strong>: Flink features a sophisticated metrics system to collect and report system and user-defined metrics. Metrics can be exported to several reporters, including <a href="https://en.wikipedia.org/wiki/Java_Management_Extensions">JMX</a>, Ganglia, <a href="https://graphiteapp.org/">Graphite</a>, <a href="https://prometheus.io/">Prometheus</a>, <a href="https://github.com/etsy/statsd">StatsD</a>, <a href="https://www.datadoghq.com/">Datadog</a>, and <a href="https://www.slf4j.org/">Slf4j</a>.</li>
+ <li><strong>REST API</strong>: Flink exposes a REST API to submit a new application, take a savepoint of a running application, or cancel an application. The REST API also exposes meta data and collected metrics of running or completed applications.</li>
+</ul>
+
+<hr />
+
+<div class="row">
+ <div class="col-sm-12" style="background-color: #f8f8f8;">
+ <h2>
+ <a href="/flink-architecture.html">Architecture</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ <a href="/flink-applications.html">Applications</a>
+ <span class="glyphicon glyphicon-chevron-right"></span>
+ Operations
+ </h2>
+ </div>
+</div>
+<hr />
+
+
+
+ </div>
+</div>
+
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2017 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/gettinghelp.html
----------------------------------------------------------------------
diff --git a/content/gettinghelp.html b/content/gettinghelp.html
new file mode 100644
index 0000000..370ae89
--- /dev/null
+++ b/content/gettinghelp.html
@@ -0,0 +1,333 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: Getting Help</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+ <!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Quickstart -->
+ <li>
+ <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">1.5 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.6 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li class="active"><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/how-to-contribute.html">How to Contribute</a></li>
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ </ul>
+
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <h1>Getting Help</h1>
+
+ <hr />
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#having-a-question" id="markdown-toc-having-a-question">Having a Question?</a> <ul>
+ <li><a href="#user-mailing-list" id="markdown-toc-user-mailing-list">User Mailing List</a></li>
+ <li><a href="#stack-overflow" id="markdown-toc-stack-overflow">Stack Overflow</a></li>
+ </ul>
+ </li>
+ <li><a href="#found-a-bug" id="markdown-toc-found-a-bug">Found a Bug?</a></li>
+ <li><a href="#got-an-error-message" id="markdown-toc-got-an-error-message">Got an Error Message?</a> <ul>
+ <li><a href="#i-have-a-notserializableexception" id="markdown-toc-i-have-a-notserializableexception">I have a NotSerializableException.</a></li>
+ <li><a href="#using-the-scala-api-i-get-an-error-about-implicit-values-and-evidence-parameters" id="markdown-toc-using-the-scala-api-i-get-an-error-about-implicit-values-and-evidence-parameters">Using the Scala API, I get an error about implicit values and evidence parameters.</a></li>
+ <li><a href="#i-see-a-classcastexception-x-cannot-be-cast-to-x" id="markdown-toc-i-see-a-classcastexception-x-cannot-be-cast-to-x">I see a ClassCastException: X cannot be cast to X.</a></li>
+ <li><a href="#i-have-an-abstractmethoderror-or-nosuchfielderror" id="markdown-toc-i-have-an-abstractmethoderror-or-nosuchfielderror">I have an AbstractMethodError or NoSuchFieldError.</a></li>
+ <li><a href="#my-datastream-application-produces-no-output-even-though-events-are-going-in" id="markdown-toc-my-datastream-application-produces-no-output-even-though-events-are-going-in">My DataStream application produces no output, even though events are going in.</a></li>
+ <li><a href="#i-see-an-exception-reporting-insufficient-number-of-network-buffers" id="markdown-toc-i-see-an-exception-reporting-insufficient-number-of-network-buffers">I see an exception reporting “Insufficient number of network buffers”.</a></li>
+ <li><a href="#my-job-fails-with-various-exceptions-from-the-hdfshadoop-code-what-can-i-do" id="markdown-toc-my-job-fails-with-various-exceptions-from-the-hdfshadoop-code-what-can-i-do">My job fails with various exceptions from the HDFS/Hadoop code. What can I do?</a></li>
+ </ul>
+ </li>
+</ul>
+
+</div>
+
+<h2 id="having-a-question">Having a Question?</h2>
+
+<p>The Apache Flink community answers many user questions every day. You can search for answers and advice in the archives or reach out to the community for help and guidance.</p>
+
+<h3 id="user-mailing-list">User Mailing List</h3>
+
+<p>Many Flink users, contributors, and committers are subscribed to Flink’s user mailing list. The user mailing list is a very good place to ask for help.</p>
+
+<p>Before posting to the mailing list, you can search the mailing list archives for email threads that discuss issues related to yours on the following websites.</p>
+
+<ul>
+ <li><a href="https://lists.apache.org/list.html?user@flink.apache.org">Apache Pony Mail Archive</a></li>
+ <li><a href="http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/">Nabble Archive</a></li>
+</ul>
+
+<p>If you’d like to post to the mailing list, you need to</p>
+
+<ol>
+ <li>subscribe to the mailing list by sending an email to <code>user-subscribe@flink.apache.org</code>,</li>
+ <li>confirm the subscription by replying to the confirmation email, and</li>
+ <li>send your email to <code>user@flink.apache.org</code>.</li>
+</ol>
+
+<p>Please note that you won’t receive a respose to your mail if you are not subscribed.</p>
+
+<h3 id="stack-overflow">Stack Overflow</h3>
+
+<p>Many members of the Flink community are active on <a href="https://stackoverflow.com">Stack Overflow</a>. You can search for questions and answers or post your questions using the <a href="https://stackoverflow.com/questions/tagged/apache-flink">[apache-flink]</a> tag.</p>
+
+<h2 id="found-a-bug">Found a Bug?</h2>
+
+<p>If you observe an unexpected behavior that might be caused by a bug, you can search for reported bugs or file a bug report in <a href="https://issues.apache.org/jira/issues/?jql=project %3D FLINK">Flink’s JIRA</a>.</p>
+
+<p>If you are unsure whether the unexpected behavior happend due to a bug or not, please post a question to the <a href="#user-mailing-list">user mailing list</a>.</p>
+
+<h2 id="got-an-error-message">Got an Error Message?</h2>
+
+<p>Identifying the cause for an error message can be challenging. In the following, we list the most common error messages and explain how to handle them.</p>
+
+<h3 id="i-have-a-notserializableexception">I have a NotSerializableException.</h3>
+
+<p>Flink uses Java serialization to distribute copies of the application logic (the functions and operations you implement,
+as well as the program configuration, etc.) to the parallel worker processes.
+Because of that, all functions that you pass to the API must be serializable, as defined by
+<a href="http://docs.oracle.com/javase/8/docs/api/java/io/Serializable.html">java.io.Serializable</a>.</p>
+
+<p>If your function is an anonymous inner class, consider the following:
+ - make the function a standalone class, or a static inner class
+ - use a Java 8 lambda function.</p>
+
+<p>Is your function is already a static class, check the fields that you assign when you create
+an instance of the class. One of the fields most likely holds a non-serializable type.
+ - In Java, use a <code>RichFunction</code> and initialize the problematic fields in the <code>open()</code> method.
+ - In Scala, you can often simply use “lazy val” to defer initialization until the distributed execution happens. This may come at a minor performance cost. You can naturally also use a <code>RichFunction</code> in Scala.</p>
+
+<h3 id="using-the-scala-api-i-get-an-error-about-implicit-values-and-evidence-parameters">Using the Scala API, I get an error about implicit values and evidence parameters.</h3>
+
+<p>This error means that the implicit value for the type information could not be provided.
+Make sure that you have an <code>import org.apache.flink.streaming.api.scala._</code> (DataStream API) or an
+<code>import org.apache.flink.api.scala._</code> (DataSet API) statement in your code.</p>
+
+<p>If you are using Flink operations inside functions or classes that take
+generic parameters, then a TypeInformation must be available for that parameter.
+This can be achieved by using a context bound:</p>
+
+<div class="highlight"><pre><code class="language-scala"><span class="k">def</span> <span class="n">myFunction</span><span class="o">[</span><span class="kt">T:</span> <span class="kt">TypeInformation</span><span class="o">](</span><span class="n">input</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span><span class="k">:</span> <span class="kt">DataSet</span><span class="o">[</span><span class="kt">Seq</span><span class="o">[</span><span class="kt">T</span><span class="o">]]</span> <span class="k">=</span> <span class="o">{</span>
+ <span class="n">input</span><span class="o">.</span><span class="n">reduceGroup</span><span class="o">(</span> <span class="n">i</span> <span class="k">=></span> <span class="n">i</span><span class="o">.</span><span class="n">toSeq</span> <span class="o">)</span>
+<span class="o">}</span></code></pre></div>
+
+<p>See <a href="http://ci.apache.org/projects/flink/flink-docs-master/dev/types_serialization.html">Type Extraction and Serialization</a> for
+an in-depth discussion of how Flink handles types.</p>
+
+<h3 id="i-see-a-classcastexception-x-cannot-be-cast-to-x">I see a ClassCastException: X cannot be cast to X.</h3>
+
+<p>When you see an exception in the style <code>com.foo.X</code> cannot be cast to <code>com.foo.X</code> (or cannot be assigned to <code>com.foo.X</code>), it means that
+multiple versions of the class <code>com.foo.X</code> have been loaded by different class loaders, and types of that class are attempted to be assigned to each other.</p>
+
+<p>The reason for that can be:</p>
+
+<ul>
+ <li>
+ <p>Class duplication through <code>child-first</code> classloading. That is an intended mechanism to allow users to use different versions of the same
+dependencies that Flink uses. However, if different copies of these classes move between Flink’s core and the user application code, such an exception
+can occur. To verify that this is the reason, try setting <code>classloader.resolve-order: parent-first</code> in the configuration.
+If that makes the error disappear, please write to the mailing list to check if that may be a bug.</p>
+ </li>
+ <li>
+ <p>Caching of classes from different execution attempts, for example by utilities like Guava’s Interners, or Avro’s Schema cache.
+Try to not use interners, or reduce the scope of the interner/cache to make sure a new cache is created whenever a new task
+execution is started.</p>
+ </li>
+</ul>
+
+<h3 id="i-have-an-abstractmethoderror-or-nosuchfielderror">I have an AbstractMethodError or NoSuchFieldError.</h3>
+
+<p>Such errors typically indicate a mix-up in some dependency version. That means a different version of a dependency (a library)
+is loaded during the execution compared to the version that code was compiled against.</p>
+
+<p>From Flink 1.4.0 on, dependencies in your application JAR file may have different versions compared to dependencies used
+by Flink’s core, or other dependencies in the classpath (for example from Hadoop). That requires <code>child-first</code> classloading
+to be activated, which is the default.</p>
+
+<p>If you see these problems in Flink 1.4+, one of the following may be true:
+ - You have a dependency version conflict within your application code. Make sure all your dependency versions are consistent.
+ - You are conflicting with a library that Flink cannot support via <code>child-first</code> classloading. Currently these are the
+ Scala standard library classes, as well as Flink’s own classes, logging APIs, and any Hadoop core classes.</p>
+
+<h3 id="my-datastream-application-produces-no-output-even-though-events-are-going-in">My DataStream application produces no output, even though events are going in.</h3>
+
+<p>If your DataStream application uses <em>Event Time</em>, check that your watermarks get updated. If no watermarks are produced,
+event time windows might never trigger, and the application would produce no results.</p>
+
+<p>You can check in Flink’s web UI (watermarks section) whether watermarks are making progress.</p>
+
+<h3 id="i-see-an-exception-reporting-insufficient-number-of-network-buffers">I see an exception reporting “Insufficient number of network buffers”.</h3>
+
+<p>If you run Flink with a very high parallelism, you may need to increase the number of network buffers.</p>
+
+<p>By default, Flink takes 10% of the JVM heap size for network buffers, with a minimum of 64MB and a maximum of 1GB.
+You can adjust all these values via <code>taskmanager.network.memory.fraction</code>, <code>taskmanager.network.memory.min</code>, and
+<code>taskmanager.network.memory.max</code>.</p>
+
+<p>Please refer to the <a href="http://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#configuring-the-network-buffers">Configuration Reference</a> for details.</p>
+
+<h3 id="my-job-fails-with-various-exceptions-from-the-hdfshadoop-code-what-can-i-do">My job fails with various exceptions from the HDFS/Hadoop code. What can I do?</h3>
+
+<p>The most common cause for that is that the Hadoop version in Flink’s classpath is different than the
+Hadoop version of the cluster you want to connect to (HDFS / YARN).</p>
+
+<p>The easiest way to fix that is to pick a Hadoop-free Flink version and simply export the Hadoop path and
+classpath from the cluster.</p>
+
+
+ </div>
+</div>
+
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2017 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/how-to-contribute.html
----------------------------------------------------------------------
diff --git a/content/how-to-contribute.html b/content/how-to-contribute.html
index a27e9d9..6335193 100644
--- a/content/how-to-contribute.html
+++ b/content/how-to-contribute.html
@@ -57,54 +57,60 @@
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-main">
- <!-- Downloads -->
- <li class=""><a class="btn btn-info" href="/downloads.html">Download Flink</a></li>
+ <!-- First menu section explains visitors what Flink is -->
- <!-- Overview -->
- <li><a href="/index.html">Home</a></li>
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
- <!-- Intro -->
- <li><a href="/introduction.html">Introduction to Flink</a></li>
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
<!-- Use cases -->
- <li><a href="/usecases.html">Flink Use Cases</a></li>
+ <li><a href="/usecases.html">Use Cases</a></li>
<!-- Powered by -->
- <li><a href="/poweredby.html">Powered by Flink</a></li>
+ <li><a href="/poweredby.html">Powered By</a></li>
- <!-- Ecosystem -->
- <li><a href="/ecosystem.html">Ecosystem</a></li>
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
- <!-- Community -->
- <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Second menu section aims to support Flink users -->
- <!-- Contribute -->
- <li class="active"><a href="/how-to-contribute.html">How to Contribute</a></li>
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
- <!-- Blog -->
- <li class=" hidden-md hidden-sm"><a href="/blog/"><b>Flink Blog</b></a></li>
+ <!-- Quickstart -->
+ <li>
+ <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">1.5 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="http://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.6 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
- <hr />
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
- <!-- Documentation -->
- <!-- <li>
- <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">Documentation <small><span class="glyphicon glyphicon-new-window"></span></small></a>
- </li> -->
- <li class="dropdown">
- <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation
- <span class="caret"></span></a>
- <ul class="dropdown-menu">
- <li><a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5" target="_blank">1.5 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
- <li><a href="http://ci.apache.org/projects/flink/flink-docs-master" target="_blank">1.6 (Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
- </ul>
- </li>
+ <!-- Third menu section aim to support community and contributors -->
- <!-- Quickstart -->
- <li>
- <a href="http://ci.apache.org/projects/flink/flink-docs-release-1.5/quickstart/setup_quickstart.html" target="_blank">Quickstart <small><span class="glyphicon glyphicon-new-window"></span></small></a>
- </li>
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Contribute -->
+ <li class="active"><a href="/how-to-contribute.html">How to Contribute</a></li>
<!-- GitHub -->
<li>
@@ -114,13 +120,9 @@
</ul>
-
<ul class="nav navbar-nav navbar-bottom">
<hr />
- <!-- FAQ -->
- <li ><a href="/faq.html">Project FAQ</a></li>
-
<!-- Twitter -->
<li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
@@ -137,7 +139,9 @@
<div class="col-sm-12">
<h1>How To Contribute</h1>
- <p>Apache Flink is developed by an open and friendly community. Everybody is cordially welcome to join the community and contribute to Apache Flink. There are several ways to interact with the community and to contribute to Flink including asking questions, filing bug reports, proposing new features, joining discussions on the mailing lists, contributing code or documentation, improving the website, or testing release candidates.</p>
+ <hr />
+
+<p>Apache Flink is developed by an open and friendly community. Everybody is cordially welcome to join the community and contribute to Apache Flink. There are several ways to interact with the community and to contribute to Flink including asking questions, filing bug reports, proposing new features, joining discussions on the mailing lists, contributing code or documentation, improving the website, or testing release candidates.</p>
<div class="page-toc">
<ul id="markdown-toc">
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/img/api-stack.png
----------------------------------------------------------------------
diff --git a/content/img/api-stack.png b/content/img/api-stack.png
new file mode 100644
index 0000000..4de2c94
Binary files /dev/null and b/content/img/api-stack.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/img/bounded-unbounded.png
----------------------------------------------------------------------
diff --git a/content/img/bounded-unbounded.png b/content/img/bounded-unbounded.png
new file mode 100644
index 0000000..29dfe8a
Binary files /dev/null and b/content/img/bounded-unbounded.png differ
http://git-wip-us.apache.org/repos/asf/flink-web/blob/df053d43/content/img/deployment-modes.png
----------------------------------------------------------------------
diff --git a/content/img/deployment-modes.png b/content/img/deployment-modes.png
new file mode 100644
index 0000000..b3f913d
Binary files /dev/null and b/content/img/deployment-modes.png differ