You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2017/01/11 20:28:22 UTC
[1/9] flink git commit: [hotfix] [docs] Move 'dev/state_backends' to
'ops/state_backends'
Repository: flink
Updated Branches:
refs/heads/master 9cbb0a9ee -> 3f7f17285
[hotfix] [docs] Move 'dev/state_backends' to 'ops/state_backends'
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/de3a35ce
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/de3a35ce
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/de3a35ce
Branch: refs/heads/master
Commit: de3a35ce84508d949b5d918da8127b412959d538
Parents: 9cbb0a9
Author: Stephan Ewen <se...@apache.org>
Authored: Mon Jan 9 19:39:44 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:10:11 2017 +0100
----------------------------------------------------------------------
docs/dev/state.md | 4 +-
docs/dev/state_backends.md | 148 ----------------------------------
docs/ops/README.md | 21 +++++
docs/ops/state_backends.md | 148 ++++++++++++++++++++++++++++++++++
docs/redirects/state_backends.md | 2 +-
docs/setup/aws.md | 4 +-
docs/setup/savepoints.md | 2 +-
7 files changed, 175 insertions(+), 154 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/dev/state.md
----------------------------------------------------------------------
diff --git a/docs/dev/state.md b/docs/dev/state.md
index 73a4ceb..4478bfc 100644
--- a/docs/dev/state.md
+++ b/docs/dev/state.md
@@ -40,7 +40,7 @@ Flink's state interface.
By default state checkpoints will be stored in-memory at the JobManager. For proper persistence of large
state, Flink supports storing the checkpoints on file systems (HDFS, S3, or any mounted POSIX file system),
which can be configured in the `flink-conf.yaml` or via `StreamExecutionEnvironment.setStateBackend(\u2026)`.
-See [state backends]({{ site.baseurl }}/dev/state_backends.html) for information
+See [state backends]({{ site.baseurl }}/ops/state_backends.html) for information
about the available state backends and how to configure them.
* ToC
@@ -52,7 +52,7 @@ Enabling Checkpointing
Flink has a checkpointing mechanism that recovers streaming jobs after failures. The checkpointing mechanism requires a *persistent* (or *durable*) source that
can be asked for prior records again (Apache Kafka is a good example of such a source).
-The checkpointing mechanism stores the progress in the data sources and data sinks, the state of windows, as well as the user-defined state (see [Working with State]({{ site.baseurl }}/dev/state.html)) consistently to provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured [state backend]({{ site.baseurl }}/dev/state_backends.html).
+The checkpointing mechanism stores the progress in the data sources and data sinks, the state of windows, as well as the user-defined state (see [Working with State]({{ site.baseurl }}/dev/state.html)) consistently to provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured [state backend]({{ site.baseurl }}/ops/state_backends.html).
The [docs on streaming fault tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html) describe in detail the technique behind Flink's streaming fault tolerance mechanism.
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/dev/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/dev/state_backends.md b/docs/dev/state_backends.md
deleted file mode 100644
index af9934d..0000000
--- a/docs/dev/state_backends.md
+++ /dev/null
@@ -1,148 +0,0 @@
----
-title: "State Backends"
-nav-parent_id: setup
-nav-pos: 11
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Programs written in the [Data Stream API]({{ site.baseurl }}/dev/datastream_api.html) often hold state in various forms:
-
-- Windows gather elements or aggregates until they are triggered
-- Transformation functions may use the key/value state interface to store values
-- Transformation functions may implement the `Checkpointed` interface to make their local variables fault tolerant
-
-See also [Working with State]({{ site.baseurl }}/dev/state.html) in the streaming API guide.
-
-When checkpointing is activated, such state is persisted upon checkpoints to guard against data loss and recover consistently.
-How the state is represented internally, and how and where it is persisted upon checkpoints depends on the
-chosen **State Backend**.
-
-* ToC
-{:toc}
-
-## Available State Backends
-
-Out of the box, Flink bundles these state backends:
-
- - *MemoryStateBackend*
- - *FsStateBackend*
- - *RocksDBStateBackend*
-
-If nothing else is configured, the system will use the MemoryStateBackend.
-
-
-### The MemoryStateBackend
-
-The *MemoryStateBackend* holds data internally as objects on the Java heap. Key/value state and window operators hold hash tables
-that store the values, triggers, etc.
-
-Upon checkpoints, this state backend will snapshot the state and send it as part of the checkpoint acknowledgement messages to the
-JobManager (master), which stores it on its heap as well.
-
-Limitations of the MemoryStateBackend:
-
- - The size of each individual state is by default limited to 5 MB. This value can be increased in the constructor of the MemoryStateBackend.
- - Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size (see [Configuration]({{ site.baseurl }}/setup/config.html)).
- - The aggregate state must fit into the JobManager memory.
-
-The MemoryStateBackend is encouraged for:
-
- - Local development and debugging
- - Jobs that do hold little state, such as jobs that consist only of record-at-a-time functions (Map, FlatMap, Filter, ...). The Kafka Consumer requires very little state.
-
-
-### The FsStateBackend
-
-The *FsStateBackend* is configured with a file system URL (type, address, path), such as "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
-
-The FsStateBackend holds in-flight data in the TaskManager's memory. Upon checkpointing, it writes state snapshots into files in the configured file system and directory. Minimal metadata is stored in the JobManager's memory (or, in high-availability mode, in the metadata checkpoint).
-
-The FsStateBackend is encouraged for:
-
- - Jobs with large state, long windows, large key/value states.
- - All high-availability setups.
-
-### The RocksDBStateBackend
-
-The *RocksDBStateBackend* is configured with a file system URL (type, address, path), such as "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
-
-The RocksDBStateBackend holds in-flight data in a [RocksDB](http://rocksdb.org) data base
-that is (per default) stored in the TaskManager data directories. Upon checkpointing, the whole
-RocksDB data base will be checkpointed into the configured file system and directory. Minimal
-metadata is stored in the JobManager's memory (or, in high-availability mode, in the metadata checkpoint).
-
-The RocksDBStateBackend is encouraged for:
-
- - Jobs with very large state, long windows, large key/value states.
- - All high-availability setups.
-
-Note that the amount of state that you can keep is only limited by the amount of disc space available.
-This allows keeping very large state, compared to the FsStateBackend that keeps state in memory.
-This also means, however, that the maximum throughput that can be achieved will be lower with
-this state backend.
-
-## Configuring a State Backend
-
-State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
-
-
-### Setting the Per-job State Backend
-
-The per-job state backend is set on the `StreamExecutionEnvironment` of the job, as shown in the example below:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"));
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment()
-env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"))
-{% endhighlight %}
-</div>
-</div>
-
-
-### Setting Default State Backend
-
-A default state backend can be configured in the `flink-conf.yaml`, using the configuration key `state.backend`.
-
-Possible values for the config entry are *jobmanager* (MemoryStateBackend), *filesystem* (FsStateBackend), or the fully qualified class
-name of the class that implements the state backend factory [FsStateBackendFactory](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackendFactory.java),
-such as `org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory` for RocksDBStateBackend.
-
-In the case where the default state backend is set to *filesystem*, the entry `state.backend.fs.checkpointdir` defines the directory where the checkpoint data will be stored.
-
-A sample section in the configuration file could look as follows:
-
-~~~
-# The backend that will be used to store operator state checkpoints
-
-state.backend: filesystem
-
-
-# Directory for storing checkpoints
-
-state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints
-~~~
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/ops/README.md
----------------------------------------------------------------------
diff --git a/docs/ops/README.md b/docs/ops/README.md
new file mode 100644
index 0000000..5fe6568
--- /dev/null
+++ b/docs/ops/README.md
@@ -0,0 +1,21 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This folder contains the documentation in the category
+**Deployment & Operations**.
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/ops/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/ops/state_backends.md b/docs/ops/state_backends.md
new file mode 100644
index 0000000..af9934d
--- /dev/null
+++ b/docs/ops/state_backends.md
@@ -0,0 +1,148 @@
+---
+title: "State Backends"
+nav-parent_id: setup
+nav-pos: 11
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Programs written in the [Data Stream API]({{ site.baseurl }}/dev/datastream_api.html) often hold state in various forms:
+
+- Windows gather elements or aggregates until they are triggered
+- Transformation functions may use the key/value state interface to store values
+- Transformation functions may implement the `Checkpointed` interface to make their local variables fault tolerant
+
+See also [Working with State]({{ site.baseurl }}/dev/state.html) in the streaming API guide.
+
+When checkpointing is activated, such state is persisted upon checkpoints to guard against data loss and recover consistently.
+How the state is represented internally, and how and where it is persisted upon checkpoints depends on the
+chosen **State Backend**.
+
+* ToC
+{:toc}
+
+## Available State Backends
+
+Out of the box, Flink bundles these state backends:
+
+ - *MemoryStateBackend*
+ - *FsStateBackend*
+ - *RocksDBStateBackend*
+
+If nothing else is configured, the system will use the MemoryStateBackend.
+
+
+### The MemoryStateBackend
+
+The *MemoryStateBackend* holds data internally as objects on the Java heap. Key/value state and window operators hold hash tables
+that store the values, triggers, etc.
+
+Upon checkpoints, this state backend will snapshot the state and send it as part of the checkpoint acknowledgement messages to the
+JobManager (master), which stores it on its heap as well.
+
+Limitations of the MemoryStateBackend:
+
+ - The size of each individual state is by default limited to 5 MB. This value can be increased in the constructor of the MemoryStateBackend.
+ - Irrespective of the configured maximal state size, the state cannot be larger than the akka frame size (see [Configuration]({{ site.baseurl }}/setup/config.html)).
+ - The aggregate state must fit into the JobManager memory.
+
+The MemoryStateBackend is encouraged for:
+
+ - Local development and debugging
+ - Jobs that do hold little state, such as jobs that consist only of record-at-a-time functions (Map, FlatMap, Filter, ...). The Kafka Consumer requires very little state.
+
+
+### The FsStateBackend
+
+The *FsStateBackend* is configured with a file system URL (type, address, path), such as "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
+
+The FsStateBackend holds in-flight data in the TaskManager's memory. Upon checkpointing, it writes state snapshots into files in the configured file system and directory. Minimal metadata is stored in the JobManager's memory (or, in high-availability mode, in the metadata checkpoint).
+
+The FsStateBackend is encouraged for:
+
+ - Jobs with large state, long windows, large key/value states.
+ - All high-availability setups.
+
+### The RocksDBStateBackend
+
+The *RocksDBStateBackend* is configured with a file system URL (type, address, path), such as "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
+
+The RocksDBStateBackend holds in-flight data in a [RocksDB](http://rocksdb.org) data base
+that is (per default) stored in the TaskManager data directories. Upon checkpointing, the whole
+RocksDB data base will be checkpointed into the configured file system and directory. Minimal
+metadata is stored in the JobManager's memory (or, in high-availability mode, in the metadata checkpoint).
+
+The RocksDBStateBackend is encouraged for:
+
+ - Jobs with very large state, long windows, large key/value states.
+ - All high-availability setups.
+
+Note that the amount of state that you can keep is only limited by the amount of disc space available.
+This allows keeping very large state, compared to the FsStateBackend that keeps state in memory.
+This also means, however, that the maximum throughput that can be achieved will be lower with
+this state backend.
+
+## Configuring a State Backend
+
+State backends can be configured per job. In addition, you can define a default state backend to be used when the
+job does not explicitly define a state backend.
+
+
+### Setting the Per-job State Backend
+
+The per-job state backend is set on the `StreamExecutionEnvironment` of the job, as shown in the example below:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"));
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment()
+env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"))
+{% endhighlight %}
+</div>
+</div>
+
+
+### Setting Default State Backend
+
+A default state backend can be configured in the `flink-conf.yaml`, using the configuration key `state.backend`.
+
+Possible values for the config entry are *jobmanager* (MemoryStateBackend), *filesystem* (FsStateBackend), or the fully qualified class
+name of the class that implements the state backend factory [FsStateBackendFactory](https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsStateBackendFactory.java),
+such as `org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory` for RocksDBStateBackend.
+
+In the case where the default state backend is set to *filesystem*, the entry `state.backend.fs.checkpointdir` defines the directory where the checkpoint data will be stored.
+
+A sample section in the configuration file could look as follows:
+
+~~~
+# The backend that will be used to store operator state checkpoints
+
+state.backend: filesystem
+
+
+# Directory for storing checkpoints
+
+state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints
+~~~
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/redirects/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/redirects/state_backends.md b/docs/redirects/state_backends.md
index f575111..3a21aaa 100644
--- a/docs/redirects/state_backends.md
+++ b/docs/redirects/state_backends.md
@@ -1,7 +1,7 @@
---
title: "State Backends"
layout: redirect
-redirect: /dev/state_backends.html
+redirect: /ops/state_backends.html
permalink: /apis/streaming/state_backends.html
---
<!--
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/setup/aws.md
----------------------------------------------------------------------
diff --git a/docs/setup/aws.md b/docs/setup/aws.md
index 83b5d97..d165955 100644
--- a/docs/setup/aws.md
+++ b/docs/setup/aws.md
@@ -57,7 +57,7 @@ HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m yarn-cluster examples/streamin
## S3: Simple Storage Service
-[Amazon Simple Storage Service](http://aws.amazon.com/s3/) (Amazon S3) provides cloud object storage for a variety of use cases. You can use S3 with Flink for **reading** and **writing data** as well in conjunction with the [streaming **state backends**]({{ site.baseurl}}/dev/state_backends.html).
+[Amazon Simple Storage Service](http://aws.amazon.com/s3/) (Amazon S3) provides cloud object storage for a variety of use cases. You can use S3 with Flink for **reading** and **writing data** as well in conjunction with the [streaming **state backends**]({{ site.baseurl}}/ops/state_backends.html).
You can use S3 objects like regular files by specifying paths in the following format:
@@ -78,7 +78,7 @@ stream.writeAsText("s3://<bucket>/<endpoint>");
env.setStateBackend(new FsStateBackend("s3://<your-bucket>/<endpoint>"));
```
-Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{ site.baseurl }}/setup/jobmanager_high_availability.html) or the [RocksDBStateBackend]({{ site.baseurl }}/dev/state_backends.html#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI.
+Note that these examples are *not* exhaustive and you can use S3 in other places as well, including your [high availability setup]({{ site.baseurl }}/setup/jobmanager_high_availability.html) or the [RocksDBStateBackend]({{ site.baseurl }}/ops/state_backends.html#the-rocksdbstatebackend); everywhere that Flink expects a FileSystem URI.
### Set S3 FileSystem
http://git-wip-us.apache.org/repos/asf/flink/blob/de3a35ce/docs/setup/savepoints.md
----------------------------------------------------------------------
diff --git a/docs/setup/savepoints.md b/docs/setup/savepoints.md
index 2866635..2a1f631 100644
--- a/docs/setup/savepoints.md
+++ b/docs/setup/savepoints.md
@@ -22,7 +22,7 @@ specific language governing permissions and limitations
under the License.
-->
-Programs written in the [Data Stream API](index.html) can resume execution from a **savepoint**. Savepoints allow both updating your programs and your Flink cluster without losing any state. This page covers all steps to trigger, restore, and dispose savepoints. For more details on how Flink handles state and failures, check out the [State in Streaming Programs]({{ site.baseurl }}/dev/state_backends.html) and [Fault Tolerance](fault_tolerance.html) pages.
+Programs written in the [Data Stream API](index.html) can resume execution from a **savepoint**. Savepoints allow both updating your programs and your Flink cluster without losing any state. This page covers all steps to trigger, restore, and dispose savepoints. For more details on how Flink handles state and failures, check out the [State in Streaming Programs]({{ site.baseurl }}/ops/state_backends.html) and [Fault Tolerance](fault_tolerance.html) pages.
* toc
{:toc}
[7/9] flink git commit: [FLINK-5458] [docs] Add a template for
migration guide
Posted by se...@apache.org.
[FLINK-5458] [docs] Add a template for migration guide
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/928238cd
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/928238cd
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/928238cd
Branch: refs/heads/master
Commit: 928238cdb62742b4b60fe197a7a36a6f33598de7
Parents: ef88af0
Author: Stephan Ewen <se...@apache.org>
Authored: Tue Jan 10 15:44:42 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:15:40 2017 +0100
----------------------------------------------------------------------
docs/dev/migration.md | 33 +++++++++++++++++++++++++++++++++
docs/index.md | 11 ++++++++++-
2 files changed, 43 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/928238cd/docs/dev/migration.md
----------------------------------------------------------------------
diff --git a/docs/dev/migration.md b/docs/dev/migration.md
new file mode 100644
index 0000000..c74952c
--- /dev/null
+++ b/docs/dev/migration.md
@@ -0,0 +1,33 @@
+---
+title: "API Migration Guides"
+nav-parent_id: dev
+nav-pos: 100
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* This will be replaced by the TOC
+{:toc}
+
+## Flink 1.1 to 1.2
+
+### State API
+
+### Fast Processing Time Window Operators
+
http://git-wip-us.apache.org/repos/asf/flink/blob/928238cd/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index 75b5328..595a094 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -29,8 +29,17 @@ Apache Flink is an open source platform for distributed stream and batch data pr
## First Steps
-- **Concepts**: Start with the basic concepts of Flink's [Dataflow Programming Model]({{ site.baseurl }}/concepts/programming-model.html) and [Distributed Runtime Environment]({{ site.baseurl }}/concepts/runtime.html). This will help you to fully understand the other parts of the documentation, including the setup and programming guides. It is highly recommended to read these sections first.
+- **Concepts**: Start with the basic concepts of Flink's [Dataflow Programming Model](concepts/programming-model.html) and [Distributed Runtime Environment](concepts/runtime.html). This will help you to fully understand the other parts of the documentation, including the setup and programming guides. It is highly recommended to read these sections first.
- **Quickstarts**: [Run an example program](quickstart/setup_quickstart.html) on your local machine or [write a simple program](quickstart/run_example_quickstart.html) working on live Wikipedia edits.
- **Programming Guides**: You can check out our guides about [basic concepts](dev/api_concepts.html) and the [DataStream API](dev/datastream_api.html) or [DataSet API](dev/batch/index.html) to learn how to write your first Flink programs.
+
+## Migration Guide
+
+For users that have used prior versions of Apache Flink we recommend checking out the [API migration guide](dev/migration.html).
+While all parts of the API that were marked as public and stable are still supported (backwards compatible), we suggest to migrate applications to the
+newer interfaces where applicable.
+
+For users that look to upgrade an operation Flink system to the latest version, we recommend to check out the guide on [upgrading Apache Flink](ops/upgrading.html)
+
[4/9] flink git commit: [hotfix] [docs] Move section about internal
snapshot implementation from 'state_backends.md' to 'stream_checkpointing.md'
Posted by se...@apache.org.
[hotfix] [docs] Move section about internal snapshot implementation from 'state_backends.md' to 'stream_checkpointing.md'
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d4a23be7
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d4a23be7
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d4a23be7
Branch: refs/heads/master
Commit: d4a23be7d296476db8dfb79636389dede1b383cd
Parents: 455c7a1
Author: Stephan Ewen <se...@apache.org>
Authored: Tue Jan 10 09:55:25 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:12:27 2017 +0100
----------------------------------------------------------------------
docs/internals/state_backends.md | 12 ------------
docs/internals/stream_checkpointing.md | 14 +++++++++++++-
2 files changed, 13 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/d4a23be7/docs/internals/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/internals/state_backends.md b/docs/internals/state_backends.md
index f6a4cc7..11d46ed 100644
--- a/docs/internals/state_backends.md
+++ b/docs/internals/state_backends.md
@@ -69,15 +69,3 @@ Examples are "ValueState", "ListState", etc. Flink's runtime encodes the states
*Raw State* is state that users and operators keep in their own data structures. When checkpointed, they only write a sequence of bytes into
the checkpoint. Flink knows nothing about the state's data structures and sees only the raw bytes.
-
-## Checkpointing Procedure
-
-When operator snapshots are taken, there are two parts: the **synchronous** and the **asynchronous** parts.
-
-Operators and state backends provide their snapshots as a Java `FutureTask`. That task contains the state where the *synchronous* part
-is completed and the *asynchronous* part is pending. The asynchronous part is then executed by a background thread for that checkpoint.
-
-Operators that checkpoint purely synchronously return an already completed `FutureTask`.
-If an asynchronous operation needs to be performed, it is executed in the `run()` method of that `FutureTask`.
-
-The tasks are cancelable, in order to release streams and other resource consuming handles.
http://git-wip-us.apache.org/repos/asf/flink/blob/d4a23be7/docs/internals/stream_checkpointing.md
----------------------------------------------------------------------
diff --git a/docs/internals/stream_checkpointing.md b/docs/internals/stream_checkpointing.md
index 75493ca..e8b3e46 100644
--- a/docs/internals/stream_checkpointing.md
+++ b/docs/internals/stream_checkpointing.md
@@ -138,7 +138,7 @@ in *at least once* mode.
Note that the above described mechanism implies that operators stop processing input records while they are storing a snapshot of their state in the *state backend*. This *synchronous* state snapshot introduces a delay every time a snapshot is taken.
-It is possible to let an operator continue processing while it stores its state snapshot, effectively letting the state snapshots happen *asynchronously* in the background. To do that, the operator must be able to produce a state object that should be stored in a way such that further modifications to the operator state do not affect that state object.
+It is possible to let an operator continue processing while it stores its state snapshot, effectively letting the state snapshots happen *asynchronously* in the background. To do that, the operator must be able to produce a state object that should be stored in a way such that further modifications to the operator state do not affect that state object. An example for that are *copy-on-write* style data structures, such as used for example in RocksDB.
After receiving the checkpoint barriers on its inputs, the operator starts the asynchronous snapshot copying of its state. It immediately emits the barrier to its outputs and continues with the regular stream processing. Once the background copy process has completed, it acknowledges the checkpoint to the checkpoint coordinator (the JobManager). The checkpoint is now only complete after all sinks received the barriers and all stateful operators acknowledged their completed backup (which may be later than the barriers reaching the sinks).
@@ -152,3 +152,15 @@ entire distributed dataflow, and gives each operator the state that was snapshot
stream from position <i>S<sub>k</sub></i>. For example in Apache Kafka, that means telling the consumer to start fetching from offset <i>S<sub>k</sub></i>.
If state was snapshotted incrementally, the operators start with the state of the latest full snapshot and then apply a series of incremental snapshot updates to that state.
+
+## Operator Snapshot Implementation
+
+When operator snapshots are taken, there are two parts: the **synchronous** and the **asynchronous** parts.
+
+Operators and state backends provide their snapshots as a Java `FutureTask`. That task contains the state where the *synchronous* part
+is completed and the *asynchronous* part is pending. The asynchronous part is then executed by a background thread for that checkpoint.
+
+Operators that checkpoint purely synchronously return an already completed `FutureTask`.
+If an asynchronous operation needs to be performed, it is executed in the `run()` method of that `FutureTask`.
+
+The tasks are cancelable, in order to release streams and other resource consuming handles.
[5/9] flink git commit: [FLINK-5456] [docs] Add stub for types of
state and state interfaces
Posted by se...@apache.org.
[FLINK-5456] [docs] Add stub for types of state and state interfaces
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/2fd543f8
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/2fd543f8
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/2fd543f8
Branch: refs/heads/master
Commit: 2fd543f80893ddd87eed773611fe7f23748008bb
Parents: d4a23be
Author: Stephan Ewen <se...@apache.org>
Authored: Tue Jan 10 12:31:21 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:12:37 2017 +0100
----------------------------------------------------------------------
docs/dev/state.md | 362 ----------------------------------
docs/dev/stream/checkpointing.md | 152 ++++++++++++++
docs/dev/stream/state.md | 78 ++++++++
docs/internals/state_backends.md | 71 -------
4 files changed, 230 insertions(+), 433 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/2fd543f8/docs/dev/state.md
----------------------------------------------------------------------
diff --git a/docs/dev/state.md b/docs/dev/state.md
deleted file mode 100644
index 4478bfc..0000000
--- a/docs/dev/state.md
+++ /dev/null
@@ -1,362 +0,0 @@
----
-title: "State & Checkpointing"
-nav-parent_id: streaming
-nav-id: state
-nav-pos: 40
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-All transformations in Flink may look like functions (in the functional processing terminology), but
-are in fact stateful operators. You can make *every* transformation (`map`, `filter`, etc) stateful
-by using Flink's state interface or checkpointing instance fields of your function. You can register
-any instance field
-as ***managed*** state by implementing an interface. In this case, and also in the case of using
-Flink's native state interface, Flink will automatically take consistent snapshots of your state
-periodically, and restore its value in the case of a failure.
-
-The end effect is that updates to any form of state are the same under failure-free execution and
-execution under failures.
-
-First, we look at how to make instance fields consistent under failures, and then we look at
-Flink's state interface.
-
-By default state checkpoints will be stored in-memory at the JobManager. For proper persistence of large
-state, Flink supports storing the checkpoints on file systems (HDFS, S3, or any mounted POSIX file system),
-which can be configured in the `flink-conf.yaml` or via `StreamExecutionEnvironment.setStateBackend(\u2026)`.
-See [state backends]({{ site.baseurl }}/ops/state_backends.html) for information
-about the available state backends and how to configure them.
-
-* ToC
-{:toc}
-
-Enabling Checkpointing
--------------------------
-
-Flink has a checkpointing mechanism that recovers streaming jobs after failures. The checkpointing mechanism requires a *persistent* (or *durable*) source that
-can be asked for prior records again (Apache Kafka is a good example of such a source).
-
-The checkpointing mechanism stores the progress in the data sources and data sinks, the state of windows, as well as the user-defined state (see [Working with State]({{ site.baseurl }}/dev/state.html)) consistently to provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured [state backend]({{ site.baseurl }}/ops/state_backends.html).
-
-The [docs on streaming fault tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html) describe in detail the technique behind Flink's streaming fault tolerance mechanism.
-
-By default, checkpointing is disabled. To enable checkpointing, call `enableCheckpointing(n)` on the `StreamExecutionEnvironment`, where *n* is the checkpoint interval in milliseconds.
-
-Other parameters for checkpointing include:
-
-- *exactly-once vs. at-least-once*: You can optionally pass a mode to the `enableCheckpointing(n)` method to choose between the two guarantee levels.
- Exactly-once is preferrable for most applications. At-least-once may be relevant for certain super-low-latency (consistently few milliseconds) applications.
-
-- *number of concurrent checkpoints*: By default, the system will not trigger another checkpoint while one is still in progress. This ensures that the topology does not spend too much time on checkpoints and not make progress with processing the streams. It is possible to allow for multiple overlapping checkpoints, which is interesting for pipelines that have a certain processing delay (for example because the functions call external services that need some time to respond) but that still want to do very frequent checkpoints (100s of milliseconds) to re-process very little upon failures.
-
-- *checkpoint timeout*: The time after which a checkpoint-in-progress is aborted, if it did not complete by then.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
-// start a checkpoint every 1000 ms
-env.enableCheckpointing(1000);
-
-// advanced options:
-
-// set mode to exactly-once (this is the default)
-env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
-
-// checkpoints have to complete within one minute, or are discarded
-env.getCheckpointConfig().setCheckpointTimeout(60000);
-
-// allow only one checkpoint to be in progress at the same time
-env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment()
-
-// start a checkpoint every 1000 ms
-env.enableCheckpointing(1000)
-
-// advanced options:
-
-// set mode to exactly-once (this is the default)
-env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
-
-// checkpoints have to complete within one minute, or are discarded
-env.getCheckpointConfig.setCheckpointTimeout(60000)
-
-// allow only one checkpoint to be in progress at the same time
-env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
-{% endhighlight %}
-</div>
-</div>
-
-{% top %}
-
-## Using the Key/Value State Interface
-
-The Key/Value state interface provides access to different types of state that are all scoped to
-the key of the current input element. This means that this type of state can only be used
-on a `KeyedStream`, which can be created via `stream.keyBy(\u2026)`.
-
-Now, we will first look at the different types of state available and then we will see
-how they can be used in a program. The available state primitives are:
-
-* `ValueState<T>`: This keeps a value that can be updated and
-retrieved (scoped to key of the input element, mentioned above, so there will possibly be one value
-for each key that the operation sees). The value can be set using `update(T)` and retrieved using
-`T value()`.
-
-* `ListState<T>`: This keeps a list of elements. You can append elements and retrieve an `Iterable`
-over all currently stored elements. Elements are added using `add(T)`, the Iterable can
-be retrieved using `Iterable<T> get()`.
-
-* `ReducingState<T>`: This keeps a single value that represents the aggregation of all values
-added to the state. The interface is the same as for `ListState` but elements added using
-`add(T)` are reduced to an aggregate using a specified `ReduceFunction`.
-
-All types of state also have a method `clear()` that clears the state for the currently
-active key (i.e. the key of the input element).
-
-It is important to keep in mind that these state objects are only used for interfacing
-with state. The state is not necessarily stored inside but might reside on disk or somewhere else.
-The second thing to keep in mind is that the value you get from the state
-depends on the key of the input element. So the value you get in one invocation of your
-user function can differ from the value in another invocation if the keys involved are different.
-
-To get a state handle you have to create a `StateDescriptor`. This holds the name of the state
-(as we will later see you can create several states, and they have to have unique names so
-that you can reference them), the type of the values that the state holds, and possibly
-a user-specified function, such as a `ReduceFunction`. Depending on what type of state you
-want to retrieve, you create either a `ValueStateDescriptor`, a `ListStateDescriptor` or
-a `ReducingStateDescriptor`.
-
-State is accessed using the `RuntimeContext`, so it is only possible in *rich functions*.
-Please see [here]({{ site.baseurl }}/dev/api_concepts#rich-functions) for
-information about that, but we will also see an example shortly. The `RuntimeContext` that
-is available in a `RichFunction` has these methods for accessing state:
-
-* `ValueState<T> getState(ValueStateDescriptor<T>)`
-* `ReducingState<T> getReducingState(ReducingStateDescriptor<T>)`
-* `ListState<T> getListState(ListStateDescriptor<T>)`
-
-This is an example `FlatMapFunction` that shows how all of the parts fit together:
-
-{% highlight java %}
-public class CountWindowAverage extends RichFlatMapFunction<Tuple2<Long, Long>, Tuple2<Long, Long>> {
-
- /**
- * The ValueState handle. The first field is the count, the second field a running sum.
- */
- private transient ValueState<Tuple2<Long, Long>> sum;
-
- @Override
- public void flatMap(Tuple2<Long, Long> input, Collector<Tuple2<Long, Long>> out) throws Exception {
-
- // access the state value
- Tuple2<Long, Long> currentSum = sum.value();
-
- // update the count
- currentSum.f0 += 1;
-
- // add the second field of the input value
- currentSum.f1 += input.f1;
-
- // update the state
- sum.update(currentSum);
-
- // if the count reaches 2, emit the average and clear the state
- if (currentSum.f0 >= 2) {
- out.collect(new Tuple2<>(input.f0, currentSum.f1 / currentSum.f0));
- sum.clear();
- }
- }
-
- @Override
- public void open(Configuration config) {
- ValueStateDescriptor<Tuple2<Long, Long>> descriptor =
- new ValueStateDescriptor<>(
- "average", // the state name
- TypeInformation.of(new TypeHint<Tuple2<Long, Long>>() {}), // type information
- Tuple2.of(0L, 0L)); // default value of the state, if nothing was set
- sum = getRuntimeContext().getState(descriptor);
- }
-}
-
-// this can be used in a streaming program like this (assuming we have a StreamExecutionEnvironment env)
-env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2.of(1L, 4L), Tuple2.of(1L, 2L))
- .keyBy(0)
- .flatMap(new CountWindowAverage())
- .print();
-
-// the printed output will be (1,4) and (1,5)
-{% endhighlight %}
-
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### State in the Scala DataStream API
-
-In addition to the interface described above, the Scala API has shortcuts for stateful
-`map()` or `flatMap()` functions with a single `ValueState` on `KeyedStream`. The user function
-gets the current value of the `ValueState` in an `Option` and must return an updated value that
-will be used to update the state.
-
-{% highlight scala %}
-val stream: DataStream[(String, Int)] = ...
-
-val counts: DataStream[(String, Int)] = stream
- .keyBy(_._1)
- .mapWithState((in: (String, Int), count: Option[Int]) =>
- count match {
- case Some(c) => ( (in._1, c), Some(c + in._2) )
- case None => ( (in._1, 0), Some(in._2) )
- })
-{% endhighlight %}
-
-## Checkpointing Instance Fields
-
-Instance fields can be checkpointed by using the `Checkpointed` interface.
-
-When the user-defined function implements the `Checkpointed` interface, the `snapshotState(\u2026)` and `restoreState(\u2026)`
-methods will be executed to draw and restore function state.
-
-In addition to that, user functions can also implement the `CheckpointListener` interface to receive notifications on
-completed checkpoints via the `notifyCheckpointComplete(long checkpointId)` method.
-Note that there is no guarantee for the user function to receive a notification if a failure happens between
-checkpoint completion and notification. The notifications should hence be treated in a way that notifications from
-later checkpoints can subsume missing notifications.
-
-The above example for `ValueState` can be implemented using instance fields like this:
-
-{% highlight java %}
-
-public class CountWindowAverage
- extends RichFlatMapFunction<Tuple2<Long, Long>, Tuple2<Long, Long>>
- implements Checkpointed<Tuple2<Long, Long>> {
-
- private Tuple2<Long, Long> sum = null;
-
- @Override
- public void flatMap(Tuple2<Long, Long> input, Collector<Tuple2<Long, Long>> out) throws Exception {
-
- // update the count
- sum.f0 += 1;
-
- // add the second field of the input value
- sum.f1 += input.f1;
-
-
- // if the count reaches 2, emit the average and clear the state
- if (sum.f0 >= 2) {
- out.collect(new Tuple2<>(input.f0, sum.f1 / sum.f0));
- sum = Tuple2.of(0L, 0L);
- }
- }
-
- @Override
- public void open(Configuration config) {
- if (sum == null) {
- // only recreate if null
- // restoreState will be called before open()
- // so this will already set the sum to the restored value
- sum = Tuple2.of(0L, 0L);
- }
- }
-
- // regularly persists state during normal operation
- @Override
- public Serializable snapshotState(long checkpointId, long checkpointTimestamp) {
- return sum;
- }
-
- // restores state on recovery from failure
- @Override
- public void restoreState(Tuple2<Long, Long> state) {
- sum = state;
- }
-}
-{% endhighlight %}
-
-## Stateful Source Functions
-
-Stateful sources require a bit more care as opposed to other operators.
-In order to make the updates to the state and output collection atomic (required for exactly-once semantics
-on failure/recovery), the user is required to get a lock from the source's context.
-
-{% highlight java %}
-public static class CounterSource
- extends RichParallelSourceFunction<Long>
- implements Checkpointed<Long> {
-
- /** current offset for exactly once semantics */
- private long offset;
-
- /** flag for job cancellation */
- private volatile boolean isRunning = true;
-
- @Override
- public void run(SourceContext<Long> ctx) {
- final Object lock = ctx.getCheckpointLock();
-
- while (isRunning) {
- // output and state update are atomic
- synchronized (lock) {
- ctx.collect(offset);
- offset += 1;
- }
- }
- }
-
- @Override
- public void cancel() {
- isRunning = false;
- }
-
- @Override
- public Long snapshotState(long checkpointId, long checkpointTimestamp) {
- return offset;
-
- }
-
- @Override
- public void restoreState(Long state) {
- offset = state;
- }
-}
-{% endhighlight %}
-
-Some operators might need the information when a checkpoint is fully acknowledged by Flink to communicate that with the outside world. In this case see the `org.apache.flink.runtime.state.CheckpointListener` interface.
-
-## State Checkpoints in Iterative Jobs
-
-Flink currently only provides processing guarantees for jobs without iterations. Enabling checkpointing on an iterative job causes an exception. In order to force checkpointing on an iterative program the user needs to set a special flag when enabling checkpointing: `env.enableCheckpointing(interval, force = true)`.
-
-Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
-
-{% top %}
-
-## Restart Strategies
-
-Flink supports different restart strategies which control how the jobs are restarted in case of a failure. For more
-information, see [Restart Strategies]({{ site.baseurl }}/dev/restart_strategies.html).
http://git-wip-us.apache.org/repos/asf/flink/blob/2fd543f8/docs/dev/stream/checkpointing.md
----------------------------------------------------------------------
diff --git a/docs/dev/stream/checkpointing.md b/docs/dev/stream/checkpointing.md
new file mode 100644
index 0000000..774d9ef
--- /dev/null
+++ b/docs/dev/stream/checkpointing.md
@@ -0,0 +1,152 @@
+---
+title: "Checkpointing"
+nav-parent_id: streaming
+nav-pos: 50
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+Every function and operator in Flink can be **stateful** (see [working with state](state.html) for details).
+Stateful functions store data across the processing of individual elements/events, making state a critical building block for
+any type of more elaborate operation.
+
+In order to make state fault tolerant, Flink needs to **checkpoint** the state. Checkpoints allow Flink to recover state and positions
+in the streams to give the application the same semantics as a failure-free execution.
+
+The [documentation on streaming fault tolerance](../../internals/stream_checkpointing.html) describe in detail the technique behind Flink's streaming fault tolerance mechanism.
+
+
+## Prerequisites
+
+Flink's checkpointing mechanism interacts with durable storage for streams and state. In general, it requires:
+
+ - A *persistent* (or *durable*) data source that can replay records for a certain amount of time. Examples for such sources are persistent messages queues (e.g., Apache Kafka, RabbitMQ, Amazon Kinesis, Google PubSub) or file systems (e.g., HDFS, S3, GFS, NFS, Ceph, ...).
+ - A persistent storage for state, typically a distributed filesystem (e.g., HDFS, S3, GFS, NFS, Ceph, ...)
+
+
+## Enabling and Configuring Checkpointing
+
+By default, checkpointing is disabled. To enable checkpointing, call `enableCheckpointing(n)` on the `StreamExecutionEnvironment`, where *n* is the checkpoint interval in milliseconds.
+
+Other parameters for checkpointing include:
+
+ - *exactly-once vs. at-least-once*: You can optionally pass a mode to the `enableCheckpointing(n)` method to choose between the two guarantee levels.
+ Exactly-once is preferrable for most applications. At-least-once may be relevant for certain super-low-latency (consistently few milliseconds) applications.
+
+ - *checkpoint timeout*: The time after which a checkpoint-in-progress is aborted, if it did not complete by then.
+
+ - *minimum time between checkpoints*: To make sure that the streaming application makes a certain amount of progress between checkpoints,
+ one can define how much time needs to pass between checkpoints. If this value is set for example to *5000*, the next checkpoint will be
+ started no sooner than 5 seconds after the previous checkpoint completed, regardless of the checkpoint duration and the checkpoint interval.
+ Note that this implies that the checkpoint interval will never be smaller than this parameter.
+
+ It is often easier to configure applications by defining the "time between checkpoints" then the checkpoint interval, because the "time between checkpoints"
+ is not susceptible to the fact that checkpoints may sometimes take longer than on average (for example if the target storage system is temporarily slow).
+
+ Note that this value also implies that the number of concurrent checkpoints is *one*.
+
+ - *number of concurrent checkpoints*: By default, the system will not trigger another checkpoint while one is still in progress.
+ This ensures that the topology does not spend too much time on checkpoints and not make progress with processing the streams.
+ It is possible to allow for multiple overlapping checkpoints, which is interesting for pipelines that have a certain processing delay
+ (for example because the functions call external services that need some time to respond) but that still want to do very frequent checkpoints
+ (100s of milliseconds) to re-process very little upon failures.
+
+ This option cannot be used when a minimum time between checkpoints is defined.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+// start a checkpoint every 1000 ms
+env.enableCheckpointing(1000);
+
+// advanced options:
+
+// set mode to exactly-once (this is the default)
+env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
+
+// make sure 500 ms of progress happen between checkpoints
+env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);
+
+// checkpoints have to complete within one minute, or are discarded
+env.getCheckpointConfig().setCheckpointTimeout(60000);
+
+// allow only one checkpoint to be in progress at the same time
+env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment()
+
+// start a checkpoint every 1000 ms
+env.enableCheckpointing(1000)
+
+// advanced options:
+
+// set mode to exactly-once (this is the default)
+env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
+
+// make sure 500 ms of progress happen between checkpoints
+env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500)
+
+// checkpoints have to complete within one minute, or are discarded
+env.getCheckpointConfig.setCheckpointTimeout(60000)
+
+// allow only one checkpoint to be in progress at the same time
+env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+
+## Selecting a State Backend
+
+The checkpointing mechanism stores the progress in the data sources and data sinks, the state of windows, as well as the [user-defined state](state.html) consistently to
+provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured
+**State Backend**.
+
+By default state will be kept in memory, and checkpoints will be stored in-memory at the master node (the JobManager). For proper persistence of large state,
+Flink supports various forms of storing and checkpointing state in so called **State Backends**, which can be set via `StreamExecutionEnvironment.setStateBackend(\u2026)`.
+
+See [state backends](../../ops/state_backends.html) for more details on the available state backends and options for job-wide and cluster-wide configuration.
+
+
+## State Checkpoints in Iterative Jobs
+
+Flink currently only provides processing guarantees for jobs without iterations. Enabling checkpointing on an iterative job causes an exception. In order to force checkpointing on an iterative program the user needs to set a special flag when enabling checkpointing: `env.enableCheckpointing(interval, force = true)`.
+
+Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.
+
+{% top %}
+
+
+## Restart Strategies
+
+Flink supports different restart strategies which control how the jobs are restarted in case of a failure. For more
+information, see [Restart Strategies]({{ site.baseurl }}/dev/restart_strategies.html).
+
+{% top %}
+
http://git-wip-us.apache.org/repos/asf/flink/blob/2fd543f8/docs/dev/stream/state.md
----------------------------------------------------------------------
diff --git a/docs/dev/stream/state.md b/docs/dev/stream/state.md
new file mode 100644
index 0000000..0b38a62
--- /dev/null
+++ b/docs/dev/stream/state.md
@@ -0,0 +1,78 @@
+---
+title: "Working with State"
+nav-parent_id: streaming
+nav-pos: 40
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+Stateful functions and operators store data across the processing of individual elements/events, making state a critical building block for
+any type of more elaborate operation. For example:
+
+ - When an application searches for certain event patterns, the state will store the sequence of events encountered so far.
+ - When aggregating events per minute, the state holds the pending aggregates.
+ - When training a machine learning model over a stream of data points, the state holds the current verstion of the model parameters.
+
+In order to make state fault tolerant, Flink needs to be aware of the state and [checkpoint](checkpointing.html) it.
+In many cases, Flink can also *manage* the state for the application, meaning Flink deals with the memory management (possibly spilling to disk
+if necessary) to allow applications to hold very large state.
+
+This document explains how to use Flink's state abstractions when developing an application.
+
+
+## Keyed State and Operator state
+
+There are two basic state backends: `Keyed State` and `Operator State`.
+
+#### Keyed State
+
+*Keyed State* is always relative to keys and can only be used in functions and operators on a `KeyedStream`.
+Examples of keyed state are the `ValueState` or `ListState` that one can create in a function on a `KeyedStream`, as
+well as the state of a keyed window operator.
+
+Keyed State is organized in so called *Key Groups*. Key Groups are the unit by which keyed state can be redistributed and
+there are as many key groups as the defined maximum parallelism.
+During execution each parallel instance of an operator gets one or more key groups.
+
+#### Operator State
+
+*Operator State* is state per parallel subtask. It subsumes the `Checkpointed` interface in Flink 1.0 and Flink 1.1.
+The new `CheckpointedFunction` interface is basically a shortcut (syntactic sugar) for the Operator State.
+
+Operator State needs special re-distribution schemes when parallelism is changed. There can be different variations of such
+schemes; the following are currently defined:
+
+ - **List-style redistribution:** Each operator returns a List of state elements. The whole state is logically a concatenation of
+ all lists. On restore/redistribution, the list is evenly divided into as many sublists as there are parallel operators.
+ Each operator gets a sublist, which can be empty, or contain one or more elements.
+
+
+## Raw and Managed State
+
+*Keyed State* and *Operator State* exist in two forms: *managed* and *raw*.
+
+*Managed State* is represented in data structures controlled by the Flink runtime, such as internal hash tables, or RocksDB.
+Examples are "ValueState", "ListState", etc. Flink's runtime encodes the states and writes them into the checkpoints.
+
+*Raw State* is state that users and operators keep in their own data structures. When checkpointed, they only write a sequence of bytes into
+the checkpoint. Flink knows nothing about the state's data structures and sees only the raw bytes.
+
http://git-wip-us.apache.org/repos/asf/flink/blob/2fd543f8/docs/internals/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/internals/state_backends.md b/docs/internals/state_backends.md
deleted file mode 100644
index 11d46ed..0000000
--- a/docs/internals/state_backends.md
+++ /dev/null
@@ -1,71 +0,0 @@
----
-title: "State and State Backends"
-nav-title: State Backends
-nav-parent_id: internals
-nav-pos: 4
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-* This will be replaced by the TOC
-{:toc}
-
-**NOTE** This document is only a sketch of some bullet points, to be fleshed out.
-
-**NOTE** The structure of State Backends changed heavily between version 1.1 and 1.2. This documentation is only applicable
-to Apache Flink version 1.2 and later.
-
-
-## Keyed State and Operator state
-
-There are two basic state backends: `Keyed State` and `Operator State`.
-
-#### Keyed State
-
-*Keyed State* is always relative to keys and can only be used in functions and operators on a `KeyedStream`.
-Examples of keyed state are the `ValueState` or `ListState` that one can create in a function on a `KeyedStream`, as
-well as the state of a keyed window operator.
-
-Keyed State is organized in so called *Key Groups*. Key Groups are the unit by which keyed state can be redistributed and
-there are as many key groups as the defined maximum parallelism.
-During execution each parallel instance of an operator gets one or more key groups.
-
-#### Operator State
-
-*Operator State* is state per parallel subtask. It subsumes the `Checkpointed` interface in Flink 1.0 and Flink 1.1.
-The new `CheckpointedFunction` interface is basically a shortcut (syntactic sugar) for the Operator State.
-
-Operator State needs special re-distribution schemes when parallelism is changed. There can be different variations of such
-schemes; the following are currently defined:
-
- - **List-style redistribution:** Each operator returns a List of state elements. The whole state is logically a concatenation of
- all lists. On restore/redistribution, the list is evenly divided into as many sublists as there are parallel operators.
- Each operator gets a sublist, which can be empty, or contain one or more elements.
-
-
-## Raw and Managed State
-
-*Keyed State* and *Operator State* exist in two forms: *managed* and *raw*.
-
-*Managed State* is represented in data structures controlled by the Flink runtime, such as internal hash tables, or RocksDB.
-Examples are "ValueState", "ListState", etc. Flink's runtime encodes the states and writes them into the checkpoints.
-
-*Raw State* is state that users and operators keep in their own data structures. When checkpointed, they only write a sequence of bytes into
-the checkpoint. Flink knows nothing about the state's data structures and sees only the raw bytes.
-
[8/9] flink git commit: [FLINK-5459] [docs] Add templates for
debugging classloading and debugging event time issues
Posted by se...@apache.org.
[FLINK-5459] [docs] Add templates for debugging classloading and debugging event time issues
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/9fb197c0
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/9fb197c0
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/9fb197c0
Branch: refs/heads/master
Commit: 9fb197c0b39e40d44ce5dd316b1f1bf3504335c6
Parents: 928238c
Author: Stephan Ewen <se...@apache.org>
Authored: Tue Jan 10 22:56:16 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:16:33 2017 +0100
----------------------------------------------------------------------
docs/monitoring/debugging_classloading.md | 45 ++++++++++++++++++++++++++
docs/monitoring/debugging_event_time.md | 37 +++++++++++++++++++++
2 files changed, 82 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/9fb197c0/docs/monitoring/debugging_classloading.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/debugging_classloading.md b/docs/monitoring/debugging_classloading.md
new file mode 100644
index 0000000..e4e908e
--- /dev/null
+++ b/docs/monitoring/debugging_classloading.md
@@ -0,0 +1,45 @@
+---
+title: "Debugging Classloading"
+nav-parent_id: monitoring
+nav-pos: 8
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+## Overview of Classloading in Flink
+
+ - What is in the Application Classloader for different deployment techs
+ - What is in the user code classloader
+
+ - Access to the user code classloader for applications
+
+## Classpath Setups
+
+ - Finding classpaths in logs
+ - Moving libraries and/or user code to the Application Classpath
+
+## Unloading of Dynamically Loaded Classes
+
+ - Checkpoint statistics overview
+ - Interpret time until checkpoints
+ - Synchronous vs. asynchronous checkpoint time
+
http://git-wip-us.apache.org/repos/asf/flink/blob/9fb197c0/docs/monitoring/debugging_event_time.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/debugging_event_time.md b/docs/monitoring/debugging_event_time.md
new file mode 100644
index 0000000..e87db0a
--- /dev/null
+++ b/docs/monitoring/debugging_event_time.md
@@ -0,0 +1,37 @@
+---
+title: "Debugging Windows & Event Time"
+nav-parent_id: monitoring
+nav-pos: 9
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+## Monitoring Current Event Time
+
+ - What metrics, how to access in the UI
+ - Event time dominated by furthest-behind source
+
+## Handling Event Time Stragglers
+
+ - Approach 1: Watermark stays late (indicated completeness), windows fire early
+ - Approach 2: Watermark heuristic with maximum lateness, windows accept late data
+
[2/9] flink git commit: [FLINK-5454] [docs] Add stub for docs on
"Tuning for large state"
Posted by se...@apache.org.
[FLINK-5454] [docs] Add stub for docs on "Tuning for large state"
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fc355f19
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fc355f19
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fc355f19
Branch: refs/heads/master
Commit: fc355f19396ec14e6deba2be5591ce085d2491f9
Parents: de3a35c
Author: Stephan Ewen <se...@apache.org>
Authored: Mon Jan 9 20:01:38 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:10:23 2017 +0100
----------------------------------------------------------------------
docs/monitoring/README.md | 21 ++++++++++
docs/monitoring/large_state_tuning.md | 62 ++++++++++++++++++++++++++++++
docs/monitoring/rest_api.md | 2 +-
3 files changed, 84 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/fc355f19/docs/monitoring/README.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/README.md b/docs/monitoring/README.md
new file mode 100644
index 0000000..88c6509
--- /dev/null
+++ b/docs/monitoring/README.md
@@ -0,0 +1,21 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This folder contains the documentation in the category
+**Debugging & Monitoring**.
http://git-wip-us.apache.org/repos/asf/flink/blob/fc355f19/docs/monitoring/large_state_tuning.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/large_state_tuning.md b/docs/monitoring/large_state_tuning.md
new file mode 100644
index 0000000..c49c106
--- /dev/null
+++ b/docs/monitoring/large_state_tuning.md
@@ -0,0 +1,62 @@
+---
+title: "Debugging and Tuning Checkpoints and Large State"
+nav-parent_id: monitoring
+nav-pos: 5
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This page gives a guide how to improve and tune applications that use large state.
+
+* ToC
+{:toc}
+
+## Monitoring State and Checkpoints
+
+ - Checkpoint statistics overview
+ - Interpret time until checkpoints
+ - Synchronous vs. asynchronous checkpoint time
+
+## Tuning Checkpointing
+
+ - Checkpoint interval
+ - Getting work done between checkpoints (min time between checkpoints)
+
+## Tuning Network Buffers
+
+ - getting a good number of buffers to use
+ - monitoring if too many buffers cause too much inflight data
+
+## Make checkpointing asynchronous where possible
+
+ - large state should be on keyed state, not operator state, because keyed state is managed, operator state not (subject to change in future versions)
+
+ - asynchronous snapshots preferrable. long synchronous snapshot times can cause problems on large state and complex topogies. move to RocksDB for that
+
+## Tuning RocksDB
+
+ - Predefined options
+ - Custom Options
+
+## Capacity planning
+
+ - Normal operation should not be constantly back pressured (link to back pressure monitor)
+ - Allow for some excess capacity to support catch-up in case of failures and checkpoint alignment skew (due to data skew or bad nodes)
+
+
http://git-wip-us.apache.org/repos/asf/flink/blob/fc355f19/docs/monitoring/rest_api.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/rest_api.md b/docs/monitoring/rest_api.md
index 2da3726..d49dece 100644
--- a/docs/monitoring/rest_api.md
+++ b/docs/monitoring/rest_api.md
@@ -1,7 +1,7 @@
---
title: "Monitoring REST API"
nav-parent_id: monitoring
-nav-pos: 3
+nav-pos: 10
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
[9/9] flink git commit: [FLINK-5460] [docs] Add placeholder for
Docker setup guide
Posted by se...@apache.org.
[FLINK-5460] [docs] Add placeholder for Docker setup guide
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/3f7f1728
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/3f7f1728
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/3f7f1728
Branch: refs/heads/master
Commit: 3f7f1728551b5caf9996c875733577db8a91ea63
Parents: 9fb197c
Author: Stephan Ewen <se...@apache.org>
Authored: Tue Jan 10 23:04:52 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:17:50 2017 +0100
----------------------------------------------------------------------
docs/setup/aws.md | 2 +-
docs/setup/deployment.md | 2 +-
docs/setup/docker.md | 29 +++++++++++++++++++++++++++++
docs/setup/gce_setup.md | 2 +-
4 files changed, 32 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/3f7f1728/docs/setup/aws.md
----------------------------------------------------------------------
diff --git a/docs/setup/aws.md b/docs/setup/aws.md
index d165955..8d04d59 100644
--- a/docs/setup/aws.md
+++ b/docs/setup/aws.md
@@ -2,7 +2,7 @@
title: "Amazon Web Services (AWS)"
nav-title: AWS
nav-parent_id: deployment
-nav-pos: 4
+nav-pos: 10
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/3f7f1728/docs/setup/deployment.md
----------------------------------------------------------------------
diff --git a/docs/setup/deployment.md b/docs/setup/deployment.md
index f3c8005..416106b 100644
--- a/docs/setup/deployment.md
+++ b/docs/setup/deployment.md
@@ -1,5 +1,5 @@
---
-title: "Deployment"
+title: "Clusters & Deployment"
nav-id: deployment
nav-parent_id: setup
nav-pos: 5
http://git-wip-us.apache.org/repos/asf/flink/blob/3f7f1728/docs/setup/docker.md
----------------------------------------------------------------------
diff --git a/docs/setup/docker.md b/docs/setup/docker.md
new file mode 100644
index 0000000..8b9757f
--- /dev/null
+++ b/docs/setup/docker.md
@@ -0,0 +1,29 @@
+---
+title: "Docker Setup"
+nav-title: Docker
+nav-parent_id: deployment
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* This will be replaced by the TOC
+{:toc}
+
+
http://git-wip-us.apache.org/repos/asf/flink/blob/3f7f1728/docs/setup/gce_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/gce_setup.md b/docs/setup/gce_setup.md
index de38ce4..f9edfcc 100644
--- a/docs/setup/gce_setup.md
+++ b/docs/setup/gce_setup.md
@@ -2,7 +2,7 @@
title: "Google Compute Engine Setup"
nav-title: Google Compute Engine
nav-parent_id: deployment
-nav-pos: 5
+nav-pos: 20
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
[6/9] flink git commit: [FLINK-5457] [docs] Add stub for Async I/O
docs
Posted by se...@apache.org.
[FLINK-5457] [docs] Add stub for Async I/O docs
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ef88af08
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ef88af08
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/ef88af08
Branch: refs/heads/master
Commit: ef88af08e95a792a8bd7b071c97c6acb2eb59668
Parents: 2fd543f
Author: Stephan Ewen <se...@apache.org>
Authored: Tue Jan 10 12:36:51 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:14:28 2017 +0100
----------------------------------------------------------------------
docs/dev/stream/asyncio.md | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/ef88af08/docs/dev/stream/asyncio.md
----------------------------------------------------------------------
diff --git a/docs/dev/stream/asyncio.md b/docs/dev/stream/asyncio.md
new file mode 100644
index 0000000..2d0867a
--- /dev/null
+++ b/docs/dev/stream/asyncio.md
@@ -0,0 +1,28 @@
+---
+title: "Async I/O for External Data Access"
+nav-parent_id: streaming
+nav-pos: 60
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+**TDB**
[3/9] flink git commit: [FLINK-5455] [docs] Add stub for Upgrading
Jobs and Framework
Posted by se...@apache.org.
[FLINK-5455] [docs] Add stub for Upgrading Jobs and Framework
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/455c7a15
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/455c7a15
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/455c7a15
Branch: refs/heads/master
Commit: 455c7a15c40289c5ff2e982ad3ddf4a9179d8a8e
Parents: fc355f1
Author: Stephan Ewen <se...@apache.org>
Authored: Mon Jan 9 20:48:35 2017 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Jan 11 21:11:35 2017 +0100
----------------------------------------------------------------------
docs/ops/upgrading.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 57 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/455c7a15/docs/ops/upgrading.md
----------------------------------------------------------------------
diff --git a/docs/ops/upgrading.md b/docs/ops/upgrading.md
new file mode 100644
index 0000000..d94e884
--- /dev/null
+++ b/docs/ops/upgrading.md
@@ -0,0 +1,57 @@
+---
+title: "Upgrading Jobs and Flink Versions"
+nav-parent_id: setup
+nav-pos: 15
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* ToC
+{:toc}
+
+## Upgrading Flink Streaming Applications
+
+ - Savepoint, stop/cancel, start from savepoint
+ - Atomic Savepoint and Stop (link to JIRA issue)
+
+ - Limitations: Breaking chaining behavior (link to Savepoint section)
+ - Encourage using `uid(...)` explicitly for every operator
+
+## Upgrading the Flink Framework Version
+
+ - Either "in place" : Savepoint -> stop/cancel -> shutdown cluster -> start new version -> start job
+ - Another cluster variant : Savepoint -> resume in other cluster -> "flip switch" -> shutdown old cluster
+
+## Compatibility Table
+
+Savepoints are compatible across Flink versions as indicated by the table below:
+
+| Created with \ Resumed With | 1.1.x | 1.2.x |
+| ---------------------------:|:-----:|:-----:|
+| 1.1.x | X | X |
+| 1.2.x | | X |
+
+
+
+## Special Considerations for Upgrades from Flink 1.1.x to Flink 1.2.x
+
+ - The parallelism of the Savepoint in Flink 1.1.x becomes the maximum parallelism in Flink 1.2.x.
+ - Increasing the parallelism for upgraded jobs is not possible out of the box.
+
+