You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by se...@apache.org on 2016/12/14 14:10:35 UTC
[1/4] flink git commit: [FLINK-5258] [docs] Reorganize the docs to
improve navigation and reduce duplication
Repository: flink
Updated Branches:
refs/heads/master e4c767a37 -> 79d7e3017
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/local_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/local_setup.md b/docs/setup/local_setup.md
deleted file mode 100644
index c23a1df..0000000
--- a/docs/setup/local_setup.md
+++ /dev/null
@@ -1,153 +0,0 @@
----
-title: "Local Setup"
-nav-title: Local
-nav-parent_id: deployment
-nav-pos: 1
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-This documentation is intended to provide instructions on how to run Flink locally on a single machine.
-
-* This will be replaced by the TOC
-{:toc}
-
-## Download
-
-Go to the [downloads page]({{ site.download_url }}) and get the ready to run package. If you want to interact with Hadoop (e.g. HDFS or HBase), make sure to pick the Flink package **matching your Hadoop version**. When in doubt or you plan to just work with the local file system pick the package for Hadoop 1.2.x.
-
-{% top %}
-
-## Requirements
-
-Flink runs on **Linux**, **Mac OS X** and **Windows**. The only requirement for a local setup is **Java 1.7.x** or higher. The following manual assumes a *UNIX-like environment*, for Windows see [Flink on Windows](#flink-on-windows).
-
-You can check the correct installation of Java by issuing the following command:
-
-~~~bash
-java -version
-~~~
-
-The command should output something comparable to the following:
-
-~~~bash
-java version "1.8.0_51"
-Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
-Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
-~~~
-
-{% top %}
-
-## Configuration
-
-**For local mode Flink is ready to go out of the box and you don't need to change the default configuration.**
-
-The out of the box configuration will use your default Java installation. You can manually set the environment variable `JAVA_HOME` or the configuration key `env.java.home` in `conf/flink-conf.yaml` if you want to manually override the Java runtime to use. Consult the [configuration page](config.html) for further details about configuring Flink.
-
-{% top %}
-
-## Starting Flink
-
-**You are now ready to start Flink.** Unpack the downloaded archive and change to the newly created `flink` directory. There you can start Flink in local mode:
-
-~~~bash
-$ tar xzf flink-*.tgz
-$ cd flink
-$ bin/start-local.sh
-Starting jobmanager.
-~~~
-
-You can check that the system is running by checking the log files in the `logs` directory:
-
-~~~bash
-$ tail log/flink-*-jobmanager-*.log
-INFO ... - Starting JobManager
-INFO ... - Starting JobManager web frontend
-INFO ... - Web frontend listening at 127.0.0.1:8081
-INFO ... - Registered TaskManager at 127.0.0.1 (akka://flink/user/taskmanager)
-~~~
-
-The JobManager will also start a web frontend on port 8081, which you can check with your browser at `http://localhost:8081`.
-
-{% top %}
-
-## Flink on Windows
-
-If you want to run Flink on Windows you need to download, unpack and configure the Flink archive as mentioned above. After that you can either use the **Windows Batch** file (`.bat`) or use **Cygwin** to run the Flink Jobmanager.
-
-### Starting with Windows Batch Files
-
-To start Flink in local mode from the *Windows Batch*, open the command window, navigate to the `bin/` directory of Flink and run `start-local.bat`.
-
-Note: The ``bin`` folder of your Java Runtime Environment must be included in Window's ``%PATH%`` variable. Follow this [guide](http://www.java.com/en/download/help/path.xml) to add Java to the ``%PATH%`` variable.
-
-~~~bash
-$ cd flink
-$ cd bin
-$ start-local.bat
-Starting Flink job manager. Web interface by default on http://localhost:8081/.
-Do not close this batch window. Stop job manager by pressing Ctrl+C.
-~~~
-
-After that, you need to open a second terminal to run jobs using `flink.bat`.
-
-{% top %}
-
-### Starting with Cygwin and Unix Scripts
-
-With *Cygwin* you need to start the Cygwin Terminal, navigate to your Flink directory and run the `start-local.sh` script:
-
-~~~bash
-$ cd flink
-$ bin/start-local.sh
-Starting jobmanager.
-~~~
-
-{% top %}
-
-### Installing Flink from Git
-
-If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similiar to this one:
-
-~~~bash
-c:/flink/bin/start-local.sh: line 30: $'\r': command not found
-~~~
-
-This error occurs, because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is, that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
-
-1. Start a Cygwin shell.
-
-2. Determine your home directory by entering
-
-~~~bash
-cd; pwd
-~~~
-
-It will return a path under the Cygwin root path.
-
-2. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you have to create it)
-
-~~~bash
-export SHELLOPTS
-set -o igncr
-~~~
-
-Save the file and open a new bash shell.
-
-{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/savepoints.md
----------------------------------------------------------------------
diff --git a/docs/setup/savepoints.md b/docs/setup/savepoints.md
index 231a4ff..2866635 100644
--- a/docs/setup/savepoints.md
+++ b/docs/setup/savepoints.md
@@ -1,7 +1,7 @@
---
title: "Savepoints"
nav-parent_id: setup
-nav-pos: 7
+nav-pos: 8
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -29,14 +29,15 @@ Programs written in the [Data Stream API](index.html) can resume execution from
## Overview
-Savepoints are **manually triggered checkpoints**, which take a snapshot of the program and write it out to a state backend. They rely on the regular checkpointing mechanism for this. During execution programs are periodically snapshotted on the worker nodes and produce checkpoints. For recovery only the last completed checkpoint is needed and older checkpoints can be safely discarded as soon as a new one is completed.
-
-Savepoints are similar to these periodic checkpoints except that they are **triggered by the user** and **don't automatically expire** when newer checkpoints are completed.
+In the example below the workers produce checkpoints **c<sub>1</sub>**, **c<sub>2</sub>**,
+**c<sub>3</sub>**, and **c<sub>4</sub>** for job *0xA312Bc*. Periodic checkpoints **c<sub>1</sub>**
+and **c<sub>3</sub>** have already been *discarded* and **c<sub>4</sub>** is the *latest
+checkpoint*. **c<sub>2</sub> is special**. It is the state associated with the savepoint
+**s<sub>1</sub>** and has been triggered by the user and it doesn't expire automatically (as
+c<sub>1</sub> and c<sub>3</sub> did after the completion of newer checkpoints).
<img src="{{ site.baseurl }}/fig/savepoints-overview.png" class="center" />
-In the above example the workers produce checkpoints **c<sub>1</sub>**, **c<sub>2</sub>**, **c<sub>3</sub>**, and **c<sub>4</sub>** for job *0xA312Bc*. Periodic checkpoints **c<sub>1</sub>** and **c<sub>3</sub>** have already been *discarded* and **c<sub>4</sub>** is the *latest checkpoint*. **c<sub>2</sub> is special**. It is the state associated with the savepoint **s<sub>1</sub>** and has been triggered by the user and it doesn't expire automatically (as c<sub>1</sub> and c<sub>3</sub> did after the completion of newer checkpoints).
-
Note that **s<sub>1</sub>** is only a **pointer to the actual checkpoint data c<sub>2</sub>**. This means that the actual state is *not copied* for the savepoint and periodic checkpoint data is kept around.
## Configuration
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/security-ssl.md
----------------------------------------------------------------------
diff --git a/docs/setup/security-ssl.md b/docs/setup/security-ssl.md
index aa99a0a..19cdcae 100644
--- a/docs/setup/security-ssl.md
+++ b/docs/setup/security-ssl.md
@@ -1,7 +1,7 @@
---
title: "SSL Setup"
nav-parent_id: setup
-nav-pos: 8
+nav-pos: 9
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/start/index.md
----------------------------------------------------------------------
diff --git a/docs/start/index.md b/docs/start/index.md
new file mode 100644
index 0000000..c6cefe2
--- /dev/null
+++ b/docs/start/index.md
@@ -0,0 +1,26 @@
+---
+section-break: true
+nav-title: '<i class="fa fa-cogs title maindish" aria-hidden="true"></i> Installation & Setup'
+title: "Installation & Setup"
+nav-id: "start"
+nav-parent_id: root
+nav-pos: 4
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
[2/4] flink git commit: [FLINK-5258] [docs] Reorganize the docs to
improve navigation and reduce duplication
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/internals/add_operator.md
----------------------------------------------------------------------
diff --git a/docs/internals/add_operator.md b/docs/internals/add_operator.md
deleted file mode 100644
index c7e26c1..0000000
--- a/docs/internals/add_operator.md
+++ /dev/null
@@ -1,253 +0,0 @@
----
-title: "How to add a new Operator"
-nav-title: "How-To: Add an Operator"
-nav-parent_id: internals
-nav-pos: 6
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Operators in the Java API can be added in multiple different ways:
-
-1. On the DataSet, as a specialization/combination of existing operators
-2. As a custom extension operator
-3. As a new runtime operator
-
-The first two approaches are typically more lightweight and easier to implement. Sometimes,
-new functionality does require a new runtime operator, or it is much more efficient to
-
-* This will be replaced by the TOC
-{:toc}
-
-## Implementing a new Operator on DataSet
-
-Many operators can be implemented as a specialization of another operator, or by means of a UDF.
-
-The simplest example are the `sum()`, `min()`, and `max()` functions on the
-{% gh_link /flink-java/src/main/java/org/apache/flink/api/java/DataSet.java "DataSet" %}.
-These functions simply call other operations with some pre-defined parameters:
-
-~~~java
-public AggregateOperator<T> sum (int field) {
- return this.aggregate (Aggregations.SUM, field);
-}
-
-~~~
-
-Some operations can be implemented as compositions of multiple other operators. An example is to implement a
-*count()* function through a combination of *map* and *aggregate*.
-
-A simple way to do this is to define a function on the {% gh_link /flink-java/src/main/java/org/apache/flink/api/java/DataSet.java "DataSet" %} that calls *map(...)* and *reduce(...)* in turn:
-
-~~~java
-public DataSet<Long> count() {
- return this.map(new MapFunction<T, Long>() {
- public Long map(T value) {
- return 1L;
- }
- })
- .reduce(new ReduceFunction<Long>() {
- public Long reduce(Long val1, Long val1) {
- return val1 + val2;
- }
- });
-}
-~~~
-
-To define a new operator without altering the DataSet class is possible by putting the functions as static members
-into another class. The example of the *count()* operator would look the following way:
-
-~~~java
-public static <T>DataSet<Long> count(DataSet<T> data) {
- return data.map(...).reduce(...);
-}
-~~~
-
-### More Complex Operators
-
-A more complex example of an operation via specialization is the {% gh_link /flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java "Aggregation Operation" %} in the Java API. It is implemented by means of a *GroupReduce* UDF.
-
-The Aggregate Operation comes with its own operator in the *Java API*, but translates itself into a {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java "GroupReduceOperatorBase" %} in the *Common API*.
-The Java API aggregation operator is only a builder that takes the types of aggregations and the field positions, and used that information to
-parameterize the GroupReduce UDF that performs the aggregations.
-
-Because the operation is translated into a GroupReduce operation, it appears as a GroupReduceOperator in the optimizer and runtime.
-
-
-
-## Implementing a Custom Extension Operator
-
-The DataSet offers a method for custom operators: `DataSet<X> runOperation(CustomUnaryOperation<T, X> operation)`.
-The *CustomUnaryOperation* interface defines operators by means of the two functions:
-
-~~~ java
-
-void setInput(DataSet<IN> inputData);
-
-DataSet<OUT> createResult();
-~~~
-
-The {% gh_link /flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/pregel/VertexCentricIteration.java "VertexCentricIteration" %} operator is implemented that way. Below is an example how to implement the *count()* operator that way.
-
-~~~ java
-public class Counter<T> implements CustomUnaryOperation<T, Long> {
-
- private DataSet<T> input;
-
- public void setInput(DataSet<IN> inputData) { this.input = inputData; }
-
- public DataSet<Long> createResult() {
- return input.map(...).reduce(...);
- }
-}
-~~~
-
-The CountOperator can be called in the following way:
-
-~~~ java
-DataSet<String> lines = ...;
-DataSet<Long> count = lines.runOperation(new Counter<String>());
-~~~
-
-
-## Implementing a new Runtime Operator
-
-Adding an new runtime operator requires changes throughout the entire stack, from the API to the runtime:
-
-- *Java API*
-- *Common API*
-- *Optimizer*
-- *Runtime*
-
-We start the description bottom up, at the example of the *mapPartition()* function, which is like a *map*
-function, but invoked only once per parallel partition.
-
-**Runtime**
-
-Runtime Operators are implemented using the {% gh_link /flink-runtime/src/main/java/org/apache/flink/runtime/operators/Driver.java "Driver" %} interface. The interface defines the methods that describe the operator towards the runtime. The {% gh_link /flink-runtime/src/main/java/org/apache/flink/runtime/operators/MapDriver.java "MapDriver" %} serves as a simple example of how those operators work.
-
-The runtime works with the `MutableObjectIterator`, which describes data streams with the ability to reuse objects, to reduce pressure on the garbage collector.
-
-An implementation of the central `run()` method for the *mapPartition* operator could look the following way:
-
-~~~ java
-public void run() throws Exception {
- final MutableObjectIterator<IN> input = this.taskContext.getInput(0);
- final MapPartitionFunction<IN, OUT> function = this.taskContext.getStub();
- final Collector<OUT> output = this.taskContext.getOutputCollector();
- final TypeSerializer<IN> serializer = this.taskContext.getInputSerializer(0);
-
- // we assume that the UDF takes a java.util.Iterator, so we wrap the MutableObjectIterator
- Iterator<IN> iterator = new MutableToRegularIteratorWrapper(input, serializer);
-
- function.mapPartition(iterator, output);
-}
-~~~
-
-To increase efficiency, it is often beneficial to implement a *chained* version of an operator. Chained
-operators run in the same thread as their preceding operator, and work with nested function calls.
-This is very efficient, because it saves serialization/deserialization overhead.
-
-To learn how to implement a chained operator, take a look at the {% gh_link /flink-runtime/src/main/java/org/apache/flink/runtime/operators/MapDriver.java "MapDriver" %} (regular) and the
-{% gh_link /flink-runtime/src/main/java/org/apache/flink/runtime/operators/chaining/ChainedMapDriver.java "ChainedMapDriver" %} (chained variant).
-
-
-### Optimizer/Compiler
-
-This section does a minimal discussion of the important steps to add an operator. Please see the {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java "Optimizer" %} for more details on how the optimizer works.
-To allow the optimizer to include a new operator in its planning, it needs a bit of information about it; in particular, the following information:
-
-- *{% gh_link /flink-runtime/src/main/java/org/apache/flink/runtime/operators/DriverStrategy.java "DriverStrategy" %}*: The operation needs to be added to the Enum, to make it available to the optimizer. The parameters to the Enum entry define which class implements the runtime operator, its chained version, whether the operator accumulates records (and needs memory for that), and whether it requires a comparator (works on keys). For our example, we can add the entry
-~~~ java
-MAP_PARTITION(MapPartitionDriver.class, null /* or chained variant */, PIPELINED, false)
-~~~
-
-- *Cost function*: The class {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/costs/CostEstimator.java "CostEstimator" %} needs to know how expensive the operation is to the system. The costs here refer to the non-UDF part of the operator. Since the operator does essentially no work (it forwards the record stream to the UDF), the costs are zero. We change the `costOperator(...)` method by adding the *MAP_PARTITION* constant to the switch statement similar to the *MAP* constant such that no cost is accounted for it.
-
-- *OperatorDescriptor*: The operator descriptors define how an operation needs to be treated by the optimizer. They describe how the operation requires the input data to be (e.g., sorted or partitioned) and that way allows the optimizer to optimize the data movement, sorting, grouping in a global fashion. They do that by describing which {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/dataproperties/RequestedGlobalProperties.java "RequestedGlobalProperties" %} (partitioning, replication, etc) and which {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/dataproperties/RequestedLocalProperties.java "RequestedLocalProperties" %} (sorting, grouping, uniqueness) the operator has, as well as how the operator affects the existing {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/dataproperties/GlobalProperties.java "GlobalProperties" %} and {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/dataproperties/LocalProp
erties.java "LocalProperties" %}. In addition, it defines a few utility methods, for example to instantiate an operator candidate.
-Since the *mapPartition()* function is very simple (no requirements on partitioning/grouping), the descriptor is very simple. Other operators have more complex requirements, for example the {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/operators/HashJoinBuildFirstProperties.java "Hash Join 1" %}, {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/operators/HashJoinBuildSecondProperties.java "Hash Join 2" %}, {% gh_link flink-optimizer/src/main/java/org/apache/flink/optimizer/operators/AbstractSortMergeJoinDescriptor.java "SortMerge Join" %}).
-The code sample below explains (with comments) how to create a descriptor for the *MapPartitionOperator*
-
- ~~~ java
- public DriverStrategy getStrategy() {
- return MAP_PARTITION;
- }
-
- // Instantiate the operator with the strategy over the input given in the form of the Channel
- public SingleInputPlanNode instantiate(Channel in, SingleInputNode node) {
- return new SingleInputPlanNode(node, "MapPartition", in, MAP_PARTITION);
- }
-
- // The operation accepts data with default global properties (arbitrary distribution)
- protected List<RequestedGlobalProperties> createPossibleGlobalProperties() {
- return Collections.singletonList(new RequestedGlobalProperties());
- }
-
- // The operation can accept data with any local properties. No grouping/sorting is necessary
- protected List<RequestedLocalProperties> createPossibleLocalProperties() {
- return Collections.singletonList(new RequestedLocalProperties());
- }
-
- // the operation itself does not affect the existing global properties.
- // The effect of the UDF's semantics// are evaluated separately (by interpreting the
- // semantic assertions)
- public GlobalProperties computeGlobalProperties(GlobalProperties gProps) {
- return gProps;
- }
-
- // since the operation can mess up all order, grouping, uniqueness, we cannot make any statements
- // about how local properties are preserved
- public LocalProperties computeLocalProperties(LocalProperties lProps) {
- return LocalProperties.EMPTY;
- }
- ~~~
-
-- *OptimizerNode*: The optimizer node is the place where all comes together. It creates the list of *OperatorDescriptors*, implements the result data set size estimation, and assigns a name to the operation. It is a relatively small class and can be more or less copied again from the {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/dag/MapNode.java "MapNode" %}.
-
-
-### Common API
-
-To make the operation available to the higher-level APIs, it needs to be added to the Common API. The simplest way to do this is to add a
-base operator. Create a class `MapPartitionOperatorBase`, after the pattern of the {% gh_link /flink-core/src/main/java/org/apache/flink/api/common/operators/base/MapOperatorBase.java "MapOperatorBase" %}.
-
-In addition, the optimizer needs to know how to create an OptimizerNode from the OperatorBase. This happens in the class
-`GraphCreatingVisitor` in the {% gh_link /flink-optimizer/src/main/java/org/apache/flink/optimizer/Optimizer.java "Optimizer" %}.
-
-*Note:* A pending idea is to allow to skip this step by unifying the OptimizerNode and the Common API operator. They essentially fulfill the
-same function. The Common API operator exists only in order for the `flink-java` and `flink-scala` packages to not have a dependency on the
-optimizer.
-
-
-### Java API
-
-Create a Java API operator that is constructed in the same way as the {% gh_link /flink-java/src/main/java/org/apache/flink/api/java/operators/MapOperator.java "MapOperator" %}. The core method is the `translateToDataFlow(...)` method, which creates the Common API operator for the Java API operator.
-
-The final step is to add a function to the `DataSet` class:
-
-~~~ java
-public <R> DataSet<R> mapPartition(MapPartitionFunction<T, R> function) {
- return new MapPartitionOperator<T, R>(this, function);
-}
-~~~
-
----
-
-*This documentation is maintained by the contributors of the individual components.
-We kindly ask anyone that adds and changes components to eventually provide a patch
-or pull request that updates these documents as well.*
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/internals/components.md
----------------------------------------------------------------------
diff --git a/docs/internals/components.md b/docs/internals/components.md
new file mode 100644
index 0000000..b750ffd
--- /dev/null
+++ b/docs/internals/components.md
@@ -0,0 +1,59 @@
+---
+title: "Component Stack"
+nav-parent_id: internals
+nav-pos: 1
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+As a software stack, Flink is a layered system. The different layers of the stack build on
+top of each other and raise the abstraction level of the program representations they accept:
+
+- The **runtime** layer receives a program in the form of a *JobGraph*. A JobGraph is a generic parallel
+data flow with arbitrary tasks that consume and produce data streams.
+
+- Both the **DataStream API** and the **DataSet API** generate JobGraphs through separate compilation
+processes. The DataSet API uses an optimizer to determine the optimal plan for the program, while
+the DataStream API uses a stream builder.
+
+- The JobGraph is executed according to a variety of deployment options available in Flink (e.g., local,
+remote, YARN, etc)
+
+- Libraries and APIs that are bundled with Flink generate DataSet or DataStream API programs. These are
+Table for queries on logical tables, FlinkML for Machine Learning, and Gelly for graph processing.
+
+You can click on the components in the figure to learn more.
+
+<center>
+ <img src="{{ site.baseurl }}/fig/stack.png" width="700px" alt="Apache Flink: Stack" usemap="#overview-stack">
+</center>
+
+<map name="overview-stack">
+<area id="lib-datastream-cep" title="CEP: Complex Event Processing" href="{{ site.baseurl }}/dev/libs/cep.html" shape="rect" coords="63,0,143,177" />
+<area id="lib-datastream-table" title="Table: Relational DataStreams" href="{{ site.baseurl }}/dev/table_api.html" shape="rect" coords="143,0,223,177" />
+<area id="lib-dataset-ml" title="FlinkML: Machine Learning" href="{{ site.baseurl }}/dev/libs/ml/index.html" shape="rect" coords="382,2,462,176" />
+<area id="lib-dataset-gelly" title="Gelly: Graph Processing" href="{{ site.baseurl }}/dev/libs/gelly/index.html" shape="rect" coords="461,0,541,177" />
+<area id="lib-dataset-table" title="Table API and SQL" href="{{ site.baseurl }}/dev/table_api.html" shape="rect" coords="544,0,624,177" />
+<area id="datastream" title="DataStream API" href="{{ site.baseurl }}/dev/datastream_api.html" shape="rect" coords="64,177,379,255" />
+<area id="dataset" title="DataSet API" href="{{ site.baseurl }}/dev/batch/index.html" shape="rect" coords="382,177,697,255" />
+<area id="runtime" title="Runtime" href="{{ site.baseurl }}/concepts/runtime.html" shape="rect" coords="63,257,700,335" />
+<area id="local" title="Local" href="{{ site.baseurl }}/quickstart/setup_quickstart.html" shape="rect" coords="62,337,275,414" />
+<area id="cluster" title="Cluster" href="{{ site.baseurl }}/setup/cluster_setup.html" shape="rect" coords="273,336,486,413" />
+<area id="cloud" title="Cloud" href="{{ site.baseurl }}/setup/gce_setup.html" shape="rect" coords="485,336,700,414" />
+</map>
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/internals/general_arch.md
----------------------------------------------------------------------
diff --git a/docs/internals/general_arch.md b/docs/internals/general_arch.md
deleted file mode 100644
index e3e8c4d..0000000
--- a/docs/internals/general_arch.md
+++ /dev/null
@@ -1,101 +0,0 @@
----
-title: "General Architecture and Process Model"
-nav-title: Architecture and Process Model
-nav-parent_id: internals
-nav-pos: 2
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-* This will be replaced by the TOC
-{:toc}
-
-## The Processes
-
-When the Flink system is started, it bring up the *JobManager* and one or more *TaskManagers*. The JobManager
-is the coordinator of the Flink system, while the TaskManagers are the workers that execute parts of the
-parallel programs. When starting the system in *local* mode, a single JobManager and TaskManager are brought
-up within the same JVM.
-
-When a program is submitted, a client is created that performs the pre-processing and turns the program
-into the parallel data flow form that is executed by the JobManager and TaskManagers. The figure below
-illustrates the different actors in the system and their interactions.
-
-<div style="text-align: center;">
-<img src="{{ site.baseurl }}/fig/process_model.svg" width="100%" alt="Flink Process Model">
-</div>
-
-## Component Stack
-
-As a software stack, Flink is a layered system. The different layers of the stack build on
-top of each other and raise the abstraction level of the program representations they accept:
-
-- The **runtime** layer receives a program in the form of a *JobGraph*. A JobGraph is a generic parallel
-data flow with arbitrary tasks that consume and produce data streams.
-
-- Both the **DataStream API** and the **DataSet API** generate JobGraphs through separate compilation
-processes. The DataSet API uses an optimizer to determine the optimal plan for the program, while
-the DataStream API uses a stream builder.
-
-- The JobGraph is executed according to a variety of deployment options available in Flink (e.g., local,
-remote, YARN, etc)
-
-- Libraries and APIs that are bundled with Flink generate DataSet or DataStream API programs. These are
-Table for queries on logical tables, FlinkML for Machine Learning, and Gelly for graph processing.
-
-You can click on the components in the figure to learn more.
-
-<center>
- <img src="{{ site.baseurl }}/fig/stack.png" width="700px" alt="Apache Flink: Stack" usemap="#overview-stack">
-</center>
-
-<map name="overview-stack">
-<area id="lib-datastream-cep" title="CEP: Complex Event Processing" href="{{ site.baseurl }}/dev/libs/cep.html" shape="rect" coords="63,0,143,177" />
-<area id="lib-datastream-table" title="Table: Relational DataStreams" href="{{ site.baseurl }}/dev/table_api.html" shape="rect" coords="143,0,223,177" />
-<area id="lib-dataset-ml" title="FlinkML: Machine Learning" href="{{ site.baseurl }}/dev/libs/ml/index.html" shape="rect" coords="382,2,462,176" />
-<area id="lib-dataset-gelly" title="Gelly: Graph Processing" href="{{ site.baseurl }}/dev/libs/gelly/index.html" shape="rect" coords="461,0,541,177" />
-<area id="lib-dataset-table" title="Table API and SQL" href="{{ site.baseurl }}/dev/table_api.html" shape="rect" coords="544,0,624,177" />
-<area id="datastream" title="DataStream API" href="{{ site.baseurl }}/dev/datastream_api.html" shape="rect" coords="64,177,379,255" />
-<area id="dataset" title="DataSet API" href="{{ site.baseurl }}/dev/batch/index.html" shape="rect" coords="382,177,697,255" />
-<area id="runtime" title="Runtime" href="{{ site.baseurl }}/internals/general_arch.html" shape="rect" coords="63,257,700,335" />
-<area id="local" title="Local" href="{{ site.baseurl }}/setup/local_setup.html" shape="rect" coords="62,337,275,414" />
-<area id="cluster" title="Cluster" href="{{ site.baseurl }}/setup/cluster_setup.html" shape="rect" coords="273,336,486,413" />
-<area id="cloud" title="Cloud" href="{{ site.baseurl }}/setup/gce_setup.html" shape="rect" coords="485,336,700,414" />
-</map>
-
-## Projects and Dependencies
-
-The Flink system code is divided into multiple sub-projects. The goal is to reduce the number of
-dependencies that a project implementing a Flink program needs, as well as to faciltate easier testing
-of smaller sub-modules.
-
-The individual projects and their dependencies are shown in the figure below.
-
-<div style="text-align: center;">
-<img src="{{ site.baseurl }}/fig/projects_dependencies.svg" alt="The Flink sub-projects and their dependencies" height="600px" style="text-align: center;"/>
-</div>
-
-In addition to the projects listed in the figure above, Flink currently contains the following sub-projects:
-
-- `flink-dist`: The *distribution* project. It defines how to assemble the compiled code, scripts, and other resources
-into the final folder structure that is ready to use.
-
-- `flink-quickstart`: Scripts, maven archetypes, and example programs for the quickstarts and tutorials.
-
-- `flink-contrib`: A series of projects that are in an early version and useful tools contributed by users. The code for the latter is maintained mainly by external contributors. The requirements for code being accepted into `flink-contrib` are lower compared to the rest of the code.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/internals/ide_setup.md
----------------------------------------------------------------------
diff --git a/docs/internals/ide_setup.md b/docs/internals/ide_setup.md
index bd931dd..1c514d0 100644
--- a/docs/internals/ide_setup.md
+++ b/docs/internals/ide_setup.md
@@ -1,7 +1,7 @@
---
title: "IDE Setup"
-nav-parent_id: internals
-nav-pos: 1
+nav-parent_id: start
+nav-pos: 3
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -25,6 +25,32 @@ under the License.
* Replaced by the TOC
{:toc}
+## IntelliJ IDEA
+
+A brief guide on how to set up IntelliJ IDEA IDE for development of the Flink core.
+As Eclipse is known to have issues with mixed Scala and Java projects, more and more contributers are migrating to IntelliJ IDEA.
+
+The following documentation describes the steps to setup IntelliJ IDEA 14.0.3 (https://www.jetbrains.com/idea/download/) with the Flink sources.
+
+Prior to doing anything, make sure that the Flink project is built at least once from the terminal:
+`mvn clean package -DskipTests`
+
+### Installing the Scala plugin
+1. Go to IntelliJ plugins settings (File -> Settings -> Plugins) and click on "Install Jetbrains plugin...".
+2. Select and install the "Scala" plugin.
+3. Restart IntelliJ
+
+### Installing the Scala compiler plugin
+1. Go to IntelliJ scala compiler settings (File -> Settings -> Build, Execution, Deployment -> Compiler -> Scala Compiler) and click on "Install Jetbrains plugin...".
+2. Click on the green plus icon on the right to add a compiler plugin
+3. Point to the paradise jar: ~/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar If there is no such file, this means that you should build Flink from the terminal as explained above.
+
+### Importing Flink
+1. Start IntelliJ IDEA and choose "Import Project"
+2. Select the root folder of the Flink repository
+3. Choose "Import project from external model" and select "Maven"
+4. Leave the default options and finish the import.
+
## Eclipse
A brief guide how to set up Eclipse for development of the Flink core.
@@ -101,28 +127,3 @@ Uncheck the box labeled "Resolve dependencies from Workspace projects", click "A
See [this post](http://stackoverflow.com/questions/25391207/how-do-i-add-execution-environment-1-8-to-eclipse-luna)
for details.
-## IntelliJ IDEA
-
-A brief guide on how to set up IntelliJ IDEA IDE for development of the Flink core.
-As Eclipse is known to have issues with mixed Scala and Java projects, more and more contributers are migrating to IntelliJ IDEA.
-
-The following documentation describes the steps to setup IntelliJ IDEA 14.0.3 (https://www.jetbrains.com/idea/download/) with the Flink sources.
-
-Prior to doing anything, make sure that the Flink project is built at least once from the terminal:
-`mvn clean package -DskipTests`
-
-### Installing the Scala plugin
-1. Go to IntelliJ plugins settings (File -> Settings -> Plugins) and click on "Install Jetbrains plugin...".
-2. Select and install the "Scala" plugin.
-3. Restart IntelliJ
-
-### Installing the Scala compiler plugin
-1. Go to IntelliJ scala compiler settings (File -> Settings -> Build, Execution, Deployment -> Compiler -> Scala Compiler) and click on "Install Jetbrains plugin...".
-2. Click on the green plus icon on the right to add a compiler plugin
-3. Point to the paradise jar: ~/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar If there is no such file, this means that you should build Flink from the terminal as explained above.
-
-### Importing Flink
-1. Start IntelliJ IDEA and choose "Import Project"
-2. Select the root folder of the Flink repository
-3. Choose "Import project from external model" and select "Maven"
-4. Leave the default options and finish the import.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/internals/index.md
----------------------------------------------------------------------
diff --git a/docs/internals/index.md b/docs/internals/index.md
index fd0af4a..8b3184c 100644
--- a/docs/internals/index.md
+++ b/docs/internals/index.md
@@ -1,9 +1,10 @@
---
title: "Internals"
nav-id: internals
-nav-pos: 5
-nav-title: '<i class="fa fa-book" aria-hidden="true"></i> Internals'
+nav-pos: 8
+nav-title: '<i class="fa fa-book title dessert" aria-hidden="true"></i> Internals'
nav-parent_id: root
+section-break: true
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/internals/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/internals/state_backends.md b/docs/internals/state_backends.md
index e9f9fd8..f6a4cc7 100644
--- a/docs/internals/state_backends.md
+++ b/docs/internals/state_backends.md
@@ -38,21 +38,21 @@ There are two basic state backends: `Keyed State` and `Operator State`.
#### Keyed State
-*Keyed State* is always relative to keys and can only be used in functions and operator on a `KeyedStream`.
+*Keyed State* is always relative to keys and can only be used in functions and operators on a `KeyedStream`.
Examples of keyed state are the `ValueState` or `ListState` that one can create in a function on a `KeyedStream`, as
-well at the state of a keyed window operator.
+well as the state of a keyed window operator.
-Keyed State is organized in so called *Key Groups*. Key Groups are the unit in which keyed state can be redistributed and
+Keyed State is organized in so called *Key Groups*. Key Groups are the unit by which keyed state can be redistributed and
there are as many key groups as the defined maximum parallelism.
-During execution, each parallel instance of an operator gets one or more key groups.
+During execution each parallel instance of an operator gets one or more key groups.
#### Operator State
-*Operator State* is state per parallel subtask. It subsume the `Checkpointed` interface in Flink 1.0 and Flink 1.1.
+*Operator State* is state per parallel subtask. It subsumes the `Checkpointed` interface in Flink 1.0 and Flink 1.1.
The new `CheckpointedFunction` interface is basically a shortcut (syntactic sugar) for the Operator State.
Operator State needs special re-distribution schemes when parallelism is changed. There can be different variations of such
-schemes, out of which the following are currently defined:
+schemes; the following are currently defined:
- **List-style redistribution:** Each operator returns a List of state elements. The whole state is logically a concatenation of
all lists. On restore/redistribution, the list is evenly divided into as many sublists as there are parallel operators.
@@ -63,22 +63,21 @@ schemes, out of which the following are currently defined:
*Keyed State* and *Operator State* exist in two forms: *managed* and *raw*.
-*Managed State* is represented in data structured controlled by the Flink runtime, such as internal hash tables, or RocksDB.
-Examples are the "ValueState", "ListState", etc. Flink's runtime encodes the states and writes them into the checkpoints.
+*Managed State* is represented in data structures controlled by the Flink runtime, such as internal hash tables, or RocksDB.
+Examples are "ValueState", "ListState", etc. Flink's runtime encodes the states and writes them into the checkpoints.
-*Raw State* is state that users and operators keep in their own data structures. Upon checkpoints, they only write a sequence or bytes into
+*Raw State* is state that users and operators keep in their own data structures. When checkpointed, they only write a sequence of bytes into
the checkpoint. Flink knows nothing about the state's data structures and sees only the raw bytes.
## Checkpointing Procedure
-When operator snapshots are takes, there are two parts: The **synchronous** and the **asynchronous** part.
+When operator snapshots are taken, there are two parts: the **synchronous** and the **asynchronous** parts.
-Operators and state backends provide their snapshots as a Java `FutureTask`. That task contains the state where tte *synchronous* part
+Operators and state backends provide their snapshots as a Java `FutureTask`. That task contains the state where the *synchronous* part
is completed and the *asynchronous* part is pending. The asynchronous part is then executed by a background thread for that checkpoint.
-Operators that checkpoint purely synchronous return an already completed `FutureTask`.
+Operators that checkpoint purely synchronously return an already completed `FutureTask`.
If an asynchronous operation needs to be performed, it is executed in the `run()` method of that `FutureTask`.
-The tasks are canceleable, in order to release streams and other resource consuming handles.
-
+The tasks are cancelable, in order to release streams and other resource consuming handles.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/monitoring/best_practices.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/best_practices.md b/docs/monitoring/best_practices.md
index 6a58a78..0bd362e 100644
--- a/docs/monitoring/best_practices.md
+++ b/docs/monitoring/best_practices.md
@@ -1,7 +1,7 @@
---
title: "Best Practices"
-nav-parent_id: monitoring
-nav-pos: 5
+nav-parent_id: dev
+nav-pos: 90
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -180,96 +180,6 @@ public static class CustomType extends Tuple11<String, String, ..., String> {
}
~~~
-
-## Register a custom serializer for your Flink program
-
-If you use a custom type in your Flink program which cannot be serialized by the
-Flink type serializer, Flink falls back to using the generic Kryo
-serializer. You may register your own serializer or a serialization system like
-Google Protobuf or Apache Thrift with Kryo. To do that, simply register the type
-class and the serializer in the `ExecutionConfig` of your Flink program.
-
-
-{% highlight java %}
-final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-// register the class of the serializer as serializer for a type
-env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, MyCustomSerializer.class);
-
-// register an instance as serializer for a type
-MySerializer mySerializer = new MySerializer();
-env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, mySerializer);
-{% endhighlight %}
-
-Note that your custom serializer has to extend Kryo's Serializer class. In the
-case of Google Protobuf or Apache Thrift, this has already been done for
-you:
-
-{% highlight java %}
-
-final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-// register the Google Protobuf serializer with Kryo
-env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, ProtobufSerializer.class);
-
-// register the serializer included with Apache Thrift as the standard serializer
-// TBaseSerializer states it should be initialized as a default Kryo serializer
-env.getConfig().addDefaultKryoSerializer(MyCustomType.class, TBaseSerializer.class);
-
-{% endhighlight %}
-
-For the above example to work, you need to include the necessary dependencies in
-your Maven project file (pom.xml). In the dependency section, add the following
-for Apache Thrift:
-
-{% highlight xml %}
-
-<dependency>
- <groupId>com.twitter</groupId>
- <artifactId>chill-thrift</artifactId>
- <version>0.5.2</version>
-</dependency>
-<!-- libthrift is required by chill-thrift -->
-<dependency>
- <groupId>org.apache.thrift</groupId>
- <artifactId>libthrift</artifactId>
- <version>0.6.1</version>
- <exclusions>
- <exclusion>
- <groupId>javax.servlet</groupId>
- <artifactId>servlet-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>httpclient</artifactId>
- </exclusion>
- </exclusions>
-</dependency>
-
-{% endhighlight %}
-
-For Google Protobuf you need the following Maven dependency:
-
-{% highlight xml %}
-
-<dependency>
- <groupId>com.twitter</groupId>
- <artifactId>chill-protobuf</artifactId>
- <version>0.5.2</version>
-</dependency>
-<!-- We need protobuf for chill-protobuf -->
-<dependency>
- <groupId>com.google.protobuf</groupId>
- <artifactId>protobuf-java</artifactId>
- <version>2.5.0</version>
-</dependency>
-
-{% endhighlight %}
-
-
-Please adjust the versions of both libraries as needed.
-
-
## Using Logback instead of Log4j
**Note: This tutorial is applicable starting from Flink 0.10**
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/monitoring/index.md
----------------------------------------------------------------------
diff --git a/docs/monitoring/index.md b/docs/monitoring/index.md
index c2582db..36399fd 100644
--- a/docs/monitoring/index.md
+++ b/docs/monitoring/index.md
@@ -1,9 +1,9 @@
---
-title: "Monitoring & Debugging"
-nav-title: '<i class="fa fa-bug" aria-hidden="true"></i> Monitoring & Debugging'
+title: "Debugging and Monitoring"
+nav-title: '<i class="fa fa-bug title maindish" aria-hidden="true"></i> Debugging & Monitoring'
nav-id: monitoring
nav-parent_id: root
-nav-pos: 4
+nav-pos: 7
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/page/css/flink.css
----------------------------------------------------------------------
diff --git a/docs/page/css/flink.css b/docs/page/css/flink.css
index 5ad0bc0..b6d4947 100644
--- a/docs/page/css/flink.css
+++ b/docs/page/css/flink.css
@@ -192,6 +192,27 @@ svg.graph text {
Side navigation
=============================================================================*/
+#sidenav a i.title {
+ min-width: 21px;
+}
+
+#sidenav a i.title.appetizer {
+ color: #FBB142;
+}
+
+#sidenav a i.title.maindish {
+ color: #7E4F89;
+}
+
+#sidenav a i.title.dessert {
+ color: #E6526F;
+}
+
+#sidenav hr.section-break {
+ margin-bottom: 5px;
+ border-top: 0px;
+}
+
#sidenav, #sidenav ul {
list-style: none;
display: block;
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/quickstart/java_api_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/java_api_quickstart.md b/docs/quickstart/java_api_quickstart.md
index 1c7bd8a..c8f21d8 100644
--- a/docs/quickstart/java_api_quickstart.md
+++ b/docs/quickstart/java_api_quickstart.md
@@ -1,8 +1,8 @@
---
-title: "Quickstart: Java API"
-nav-title: Java API
-nav-parent_id: quickstarts
-nav-pos: 3
+title: "Sample Project using the Java API"
+nav-title: Sample Project in Java
+nav-parent_id: start
+nav-pos: 0
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -64,8 +64,7 @@ There will be a new directory in your working directory. If you've used the _cur
The sample project is a __Maven project__, which contains four classes. _StreamingJob_ and _BatchJob_ are basic skeleton programs, _SocketTextStreamWordCount_ is a working streaming example and _WordCountJob_ is a working batch example. Please note that the _main_ method of all classes allow you to start Flink in a development/testing mode.
-We recommend you __import this project into your IDE__ to develop and test it. If you use Eclipse, the [m2e plugin](http://www.eclipse.org/m2e/) allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/reference/creating-sect-importing-projects.html#fig-creating-import). Some Eclipse bundles include that plugin by default, others require you to install it manually. The IntelliJ IDE also supports Maven projects out of the box.
-
+We recommend you [import this project into your IDE]({{ site.baseurl }}/internals/ide_setup) to develop and test it.
A note to Mac OS X users: The default JVM heapsize for Java is too small for Flink. You have to manually increase it. Choose "Run Configurations" -> Arguments and write into the "VM Arguments" box: "-Xmx800m" in Eclipse.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/quickstart/run_example_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/run_example_quickstart.md b/docs/quickstart/run_example_quickstart.md
index a64e87f..9ce4c84 100644
--- a/docs/quickstart/run_example_quickstart.md
+++ b/docs/quickstart/run_example_quickstart.md
@@ -1,8 +1,8 @@
---
title: "Monitoring the Wikipedia Edit Stream"
nav-title: Monitoring Wikipedia Edits
-nav-parent_id: quickstarts
-nav-pos: 2
+nav-parent_id: examples
+nav-pos: 10
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -278,8 +278,8 @@ was produced.
This should get you started with writing your own Flink programs. To learn more
you can check out our guides
-about [basic concepts]({{ site.baseurl }}/apis/common/index.html) and the
-[DataStream API]({{ site.baseurl }}/apis/streaming/index.html). Stick
+about [basic concepts]({{ site.baseurl }}/dev/api_concepts) and the
+[DataStream API]({{ site.baseurl }}/dev/datastream_api). Stick
around for the bonus exercise if you want to learn about setting up a Flink cluster on
your own machine and writing results to [Kafka](http://kafka.apache.org).
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/quickstart/scala_api_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/scala_api_quickstart.md b/docs/quickstart/scala_api_quickstart.md
index 22562f4..388f0a7 100644
--- a/docs/quickstart/scala_api_quickstart.md
+++ b/docs/quickstart/scala_api_quickstart.md
@@ -1,8 +1,8 @@
---
-title: "Quickstart: Scala API"
-nav-title: Scala API
-nav-parent_id: quickstarts
-nav-pos: 4
+title: "Sample Project using the Scala API"
+nav-title: Sample Project in Scala
+nav-parent_id: start
+nav-pos: 1
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/quickstart/setup_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/quickstart/setup_quickstart.md b/docs/quickstart/setup_quickstart.md
index 3dd0950..6aab686 100644
--- a/docs/quickstart/setup_quickstart.md
+++ b/docs/quickstart/setup_quickstart.md
@@ -1,8 +1,8 @@
---
-title: "Quickstart: Setup"
-nav-title: Setup
-nav-parent_id: quickstarts
-nav-pos: 1
+title: "Quickstart"
+nav-title: '<i class="fa fa-power-off title appetizer" aria-hidden="true"></i> Quickstart'
+nav-parent_id: root
+nav-pos: 2
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -28,13 +28,29 @@ under the License.
Get a Flink example program up and running in a few simple steps.
-## Setup: Download and Start
+## Setup: Download and Start Flink
-Flink runs on __Linux, Mac OS X, and Windows__. To be able to run Flink, the only requirement is to have a working __Java 7.x__ (or higher) installation. Windows users, please take a look at the [Flink on Windows]({{ site.baseurl }}/setup/local_setup.html#flink-on-windows) guide which describes how to run Flink on Windows for local setups.
+Flink runs on __Linux, Mac OS X, and Windows__. To be able to run Flink, the only requirement is to have a working __Java 7.x__ (or higher) installation. Windows users, please take a look at the [Flink on Windows]({{ site.baseurl }}/setup/flink_on_windows) guide which describes how to run Flink on Windows for local setups.
+
+You can check the correct installation of Java by issuing the following command:
+
+~~~bash
+java -version
+~~~
+
+If you have Java 8, the output will look something like this:
+
+~~~bash
+java version "1.8.0_111"
+Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
+Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
+~~~
### Download
-Download a binary from the [downloads page](http://flink.apache.org/downloads.html). You can pick any Hadoop/Scala combination you like.
+Download a binary from the [downloads page](http://flink.apache.org/downloads.html). You can pick
+any Hadoop/Scala combination you like. If you plan to just use the local file system, any Hadoop
+version will work fine.
### Start a Local Flink Cluster
@@ -53,9 +69,135 @@ Check the __JobManager's web frontend__ at [http://localhost:8081](http://localh
<a href="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-1.png" ><img class="img-responsive" src="{{ site.baseurl }}/page/img/quickstart-setup/jobmanager-1.png" alt="JobManager: Overview"/></a>
-## Run Example
+You can also verify that the system is running by checking the log files in the `logs` directory:
+
+~~~bash
+$ tail log/flink-*-jobmanager-*.log
+INFO ... - Starting JobManager
+INFO ... - Starting JobManager web frontend
+INFO ... - Web frontend listening at 127.0.0.1:8081
+INFO ... - Registered TaskManager at 127.0.0.1 (akka://flink/user/taskmanager)
+~~~
-Now, we are going to run the [SocketWindowWordCount example](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java) and read text from a socket and count the number of distinct words.
+## Read the Code
+
+You can find the complete source code for this SocketWindowWordCount example in [scala](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/socket/SocketWindowWordCount.scala) and [java](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/socket/SocketWindowWordCount.java) on GitHub.
+
+<div class="codetabs" markdown="1">
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+object SocketWindowWordCount {
+
+ def main(args: Array[String]) : Unit = {
+
+ // the port to connect to
+ val port: Int = try {
+ ParameterTool.fromArgs(args).getInt("port")
+ } catch {
+ case e: Exception => {
+ System.err.println("No port specified. Please run 'SocketWindowWordCount --port <port>'")
+ return
+ }
+ }
+
+ // get the execution environment
+ val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment
+
+ // get input data by connecting to the socket
+ val text = env.socketTextStream("localhost", port, '\n')
+
+ // parse the data, group it, window it, and aggregate the counts
+ val windowCounts = text
+ .flatMap { w => w.split("\\s") }
+ .map { w => WordWithCount(w, 1) }
+ .keyBy("word")
+ .timeWindow(Time.seconds(5), Time.seconds(1))
+ .sum("count")
+
+ // print the results with a single thread, rather than in parallel
+ windowCounts.print().setParallelism(1)
+
+ env.execute("Socket Window WordCount")
+ }
+
+ // Data type for words with count
+ case class WordWithCount(word: String, count: Long)
+}
+{% endhighlight %}
+</div>
+<div data-lang="java" markdown="1">
+{% highlight java %}
+public class SocketWindowWordCount {
+
+ public static void main(String[] args) throws Exception {
+
+ // the port to connect to
+ final int port;
+ try {
+ final ParameterTool params = ParameterTool.fromArgs(args);
+ port = params.getInt("port");
+ } catch (Exception e) {
+ System.err.println("No port specified. Please run 'SocketWindowWordCount --port <port>'");
+ return;
+ }
+
+ // get the execution environment
+ final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ // get input data by connecting to the socket
+ DataStream<String> text = env.socketTextStream("localhost", port, "\n");
+
+ // parse the data, group it, window it, and aggregate the counts
+ DataStream<WordWithCount> windowCounts = text
+ .flatMap(new FlatMapFunction<String, WordWithCount>() {
+ @Override
+ public void flatMap(String value, Collector<WordWithCount> out) {
+ for (String word : value.split("\\s")) {
+ out.collect(new WordWithCount(word, 1L));
+ }
+ }
+ })
+ .keyBy("word")
+ .timeWindow(Time.seconds(5), Time.seconds(1))
+ .reduce(new ReduceFunction<WordWithCount>() {
+ @Override
+ public WordWithCount reduce(WordWithCount a, WordWithCount b) {
+ return new WordWithCount(a.word, a.count + b.count);
+ }
+ });
+
+ // print the results with a single thread, rather than in parallel
+ windowCounts.print().setParallelism(1);
+
+ env.execute("Socket Window WordCount");
+ }
+
+ // Data type for words with count
+ public static class WordWithCount {
+
+ public String word;
+ public long count;
+
+ public WordWithCount() {}
+
+ public WordWithCount(String word, long count) {
+ this.word = word;
+ this.count = count;
+ }
+
+ @Override
+ public String toString() {
+ return word + " : " + count;
+ }
+ }
+}
+{% endhighlight %}
+</div>
+</div>
+
+## Run the Example
+
+Now, we are going to run this Flink application. It will read text from a socket and once a second print the number of occurances of each distinct word during the previous 5 seconds.
* First of all, we use **netcat** to start local server via
@@ -119,67 +261,4 @@ Now, we are going to run the [SocketWindowWordCount example](https://github.com/
## Next Steps
-Check out the [step-by-step example](run_example_quickstart.html) in order to get a first feel of Flink's programming APIs. When you are done with that, go ahead and read the [streaming guide]({{ site.baseurl }}/dev/datastream_api.html).
-
-### Cluster Setup
-
-__Running Flink on a cluster__ is as easy as running it locally. Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes lets you use our scripts to control
-everything.
-
-1. Copy the unpacked __flink__ directory from the downloaded archive to the same file system path
-on each node of your setup.
-2. Choose a __master node__ (JobManager) and set the `jobmanager.rpc.address` key in
-`conf/flink-conf.yaml` to its IP or hostname. Make sure that all nodes in your cluster have the same
-`jobmanager.rpc.address` configured.
-3. Add the IPs or hostnames (one per line) of all __worker nodes__ (TaskManager) to the slaves files
-in `conf/slaves`.
-
-You can now __start the cluster__ at your master node with `bin/start-cluster.sh`.
-
-The following __example__ illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files, which need to be accessible at the same path on all machines:
-
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{{ site.baseurl }}/page/img/quickstart_cluster.png" style="width: 85%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/slaves</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
-
-Have a look at the [Configuration]({{ site.baseurl }}/setup/config.html) section of the documentation to see other available configuration options.
-For Flink to run efficiently, a few configuration values need to be set.
-
-In particular,
-
- * the amount of available memory per TaskManager (`taskmanager.heap.mb`),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`taskmanager.tmp.dirs`)
-
-
-are very important configuration values.
-
-### Flink on YARN
-
-You can easily deploy Flink on your existing __YARN cluster__.
-
-1. Download the __Flink Hadoop2 package__ from the [downloads page](https://flink.apache.org/downloads.html)
-2. Make sure your __HADOOP_HOME__ (or _YARN_CONF_DIR_ or _HADOOP_CONF_DIR_) __environment variable__ is set to read your YARN and HDFS configuration.
-3. Run the __YARN client__ with: `./bin/yarn-session.sh`. You can run the client with options `-n 10 -tm 8192` to allocate 10 TaskManagers with 8GB of memory each.
+Check out some more [examples]({{ site.baseurl }}/examples) to get a better feel for Flink's programming APIs. When you are done with that, go ahead and read the [streaming guide]({{ site.baseurl }}/dev/datastream_api.html).
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/redirects/concepts.md
----------------------------------------------------------------------
diff --git a/docs/redirects/concepts.md b/docs/redirects/concepts.md
index 82790e8..5a96bf4 100644
--- a/docs/redirects/concepts.md
+++ b/docs/redirects/concepts.md
@@ -1,7 +1,7 @@
---
title: "Concepts"
layout: redirect
-redirect: /concepts/index.html
+redirect: /concepts/programming-model.html
permalink: /concepts/concepts.html
---
<!--
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/building.md
----------------------------------------------------------------------
diff --git a/docs/setup/building.md b/docs/setup/building.md
index 8c60997..8a71fc6 100644
--- a/docs/setup/building.md
+++ b/docs/setup/building.md
@@ -1,7 +1,7 @@
---
-title: Build Flink from Source
-nav-parent_id: setup
-nav-pos: 1
+title: Building Flink from Source
+nav-parent_id: start
+nav-pos: 20
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/checkpoints.md
----------------------------------------------------------------------
diff --git a/docs/setup/checkpoints.md b/docs/setup/checkpoints.md
new file mode 100644
index 0000000..ea2f406
--- /dev/null
+++ b/docs/setup/checkpoints.md
@@ -0,0 +1,57 @@
+---
+title: "Checkpoints"
+nav-parent_id: setup
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+* toc
+{:toc}
+
+## Overview
+
+TBD
+
+
+## Externalized Checkpoints
+
+You can configure periodic checkpoints to be persisted externally. Externalized checkpoints write their meta data out to persistent storage and are *not* automatically cleaned up when the job fails. This way, you will have a checkpoint around to resume from if your job fails.
+
+```java
+CheckpoingConfig config = env.getCheckpointConfig();
+config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
+```
+
+The `ExternalizedCheckpointCleanup` mode configures what happens with externalized checkpoints when you cancel the job:
+
+- **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the externalized checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
+
+- **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the externalized checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
+
+The **target directory** for the checkpoint is determined from the default checkpoint directory configuration. This is configured via the configuration key `state.checkpoints.dir`, which should point to the desired target directory:
+
+```
+state.checkpoints.dir: hdfs:///checkpoints/
+```
+
+This directory will then contain the checkpoint meta data required to restore the checkpoint. The actual checkpoint files will still be available in their configured directory. You currently can only set this via the configuration files.
+
+Follow the [savepoint guide]({{ site.baseurl }}/setup/cli.html#savepoints) when you want to resume from a specific checkpoint.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/cluster_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/cluster_setup.md b/docs/setup/cluster_setup.md
index 77e1a3c..7d3684f 100644
--- a/docs/setup/cluster_setup.md
+++ b/docs/setup/cluster_setup.md
@@ -39,6 +39,10 @@ Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **
If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+Having __passwordless SSH__ and
+__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
+everything.
+
{% top %}
### `JAVA_HOME` Configuration
@@ -51,7 +55,7 @@ You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
## Flink Setup
-Go to the [downloads page]({{ site.download_url }}) and get the ready to run package. Make sure to pick the Flink package **matching your Hadoop version**. If you don't plan to use Hadoop, pick any version.
+Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package. Make sure to pick the Flink package **matching your Hadoop version**. If you don't plan to use Hadoop, pick any version.
After downloading the latest release, copy the archive to your master node and extract it:
@@ -64,29 +68,45 @@ cd flink-*
After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
-Set the `jobmanager.rpc.address` key to point to your master node. Furthermode define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the `jobmanager.heap.mb` and `taskmanager.heap.mb` keys.
-
-The value is given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting an environment variable `FLINK_TM_HEAP` on the respective node.
-
-Finally you must provide a list of all nodes in your cluster which shall be used as worker nodes. Therefore, similar to the HDFS configuration, edit the file *conf/slaves* and enter the IP/host name of each worker node. Each worker node will later run a TaskManager.
-
-Each entry must be separated by a new line, as in the following example:
-
-~~~
-192.168.0.100
-192.168.0.101
-.
-.
-.
-192.168.0.150
-~~~
-
-The Flink directory must be available on every worker under the same path. You can use a shared NSF directory, or copy the entire Flink directory to every worker node.
+Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the `jobmanager.heap.mb` and `taskmanager.heap.mb` keys.
+
+These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting the environment variable `FLINK_TM_HEAP` on those specific nodes.
+
+Finally, you must provide a list of all nodes in your cluster which shall be used as worker nodes. Therefore, similar to the HDFS configuration, edit the file *conf/slaves* and enter the IP/host name of each worker node. Each worker node will later run a TaskManager.
+
+The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
+to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
+configuration files (which need to be accessible at the same path on all machines):
+
+<div class="row">
+ <div class="col-md-6 text-center">
+ <img src="{{ site.baseurl }}/page/img/quickstart_cluster.png" style="width: 60%">
+ </div>
+<div class="col-md-6">
+ <div class="row">
+ <p class="lead text-center">
+ /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
+ <pre>jobmanager.rpc.address: 10.0.0.1</pre>
+ </p>
+ </div>
+<div class="row" style="margin-top: 1em;">
+ <p class="lead text-center">
+ /path/to/<strong>flink/<br>conf/slaves</strong>
+ <pre>
+10.0.0.2
+10.0.0.3</pre>
+ </p>
+</div>
+</div>
+</div>
+
+The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
Please see the [configuration page](config.html) for details and additional configuration options.
In particular,
+ * the amount of available memory per JobManager (`jobmanager.heap.mb`),
* the amount of available memory per TaskManager (`taskmanager.heap.mb`),
* the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
* the total number of CPUs in the cluster (`parallelism.default`) and
@@ -126,6 +146,6 @@ bin/jobmanager.sh (start cluster)|stop|stop-all
bin/taskmanager.sh start|stop|stop-all
~~~
-Make sure to call these scripts on the hosts, on which you want to start/stop the respective instance.
+Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/config.md
----------------------------------------------------------------------
diff --git a/docs/setup/config.md b/docs/setup/config.md
index c24065a..89e8207 100644
--- a/docs/setup/config.md
+++ b/docs/setup/config.md
@@ -1,7 +1,8 @@
---
title: "Configuration"
+nav-id: "config"
nav-parent_id: setup
-nav-pos: 3
+nav-pos: 1
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -22,7 +23,9 @@ specific language governing permissions and limitations
under the License.
-->
-The default configuration parameters allow Flink to run out-of-the-box in single node setups.
+**For single-node setups Flink is ready to go out of the box and you don't need to change the default configuration to get started.**
+
+The out of the box configuration will use your default Java installation. You can manually set the environment variable `JAVA_HOME` or the configuration key `env.java.home` in `conf/flink-conf.yaml` if you want to manually override the Java runtime to use.
This page lists the most common options that are typically needed to set up a well performing (distributed) installation. In addition a full list of all available configuration parameters is listed here.
@@ -56,7 +59,7 @@ The configuration files for the TaskManagers can be different, Flink does not as
- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: 1). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores). [More about task slots](config.html#configuring-taskmanager-processing-slots).
- `parallelism.default`: The default parallelism to use for programs that have no parallelism specified. (DEFAULT: 1). For setups that have no concurrent jobs running, setting this value to NumTaskManagers * NumSlotsPerTaskManager will cause the system to use all available execution resources for the program's execution. **Note**: The default parallelism can be overwriten for an entire job by calling `setParallelism(int parallelism)` on the `ExecutionEnvironment` or by passing `-p <parallelism>` to the Flink Command-line frontend. It can be overwritten for single transformations by calling `setParallelism(int
-parallelism)` on an operator. See the [Basic API Concepts]({{site.baseurl}}/dev/api_concepts.html#parallel-execution) for more information about the parallelism.
+parallelism)` on an operator. See [Parallel Execution]({{site.baseurl}}/dev/parallel) for more information about parallelism.
- `fs.default-scheme`: The default filesystem scheme to be used, with the necessary authority to contact, e.g. the host:port of the NameNode in the case of HDFS (if needed).
By default, this is set to `file:///` which points to the local filesystem. This means that the local
@@ -534,6 +537,6 @@ Flink executes a program in parallel by splitting it into subtasks and schedulin
Each Flink TaskManager provides processing slots in the cluster. The number of slots is typically proportional to the number of available CPU cores __of each__ TaskManager. As a general recommendation, the number of available CPU cores is a good default for `taskmanager.numberOfTaskSlots`.
-When starting a Flink application, users can supply the default number of slots to use for that job. The command line value therefore is called `-p` (for parallelism). In addition, it is possible to [set the number of slots in the programming APIs]({{site.baseurl}}/dev/api_concepts.html#parallel-execution) for the whole application and individual operators.
+When starting a Flink application, users can supply the default number of slots to use for that job. The command line value therefore is called `-p` (for parallelism). In addition, it is possible to [set the number of slots in the programming APIs]({{site.baseurl}}/dev/parallel) for the whole application and for individual operators.
<img src="{{ site.baseurl }}/fig/slots_parallelism.svg" class="img-responsive" />
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/fault_tolerance.md
----------------------------------------------------------------------
diff --git a/docs/setup/fault_tolerance.md b/docs/setup/fault_tolerance.md
index cef746e..500e91a 100644
--- a/docs/setup/fault_tolerance.md
+++ b/docs/setup/fault_tolerance.md
@@ -1,8 +1,7 @@
---
-title: "Failure & Recovery Model"
-nav-id: fault_tolerance
-nav-parent_id: setup
-nav-pos: 4
+title: "Restart Strategies"
+nav-parent_id: execution
+nav-pos: 50
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -23,222 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-Flink's fault tolerance mechanism recovers programs in the presence of failures and
-continues to execute them. Such failures include machine hardware failures, network failures,
-transient program failures, etc.
-
-* This will be replaced by the TOC
-{:toc}
-
-
-Streaming Fault Tolerance
--------------------------
-
-Flink has a checkpointing mechanism that recovers streaming jobs after failures. The checkpointing mechanism requires a *persistent* (or *durable*) source that
-can be asked for prior records again (Apache Kafka is a good example of such a source).
-
-The checkpointing mechanism stores the progress in the data sources and data sinks, the state of windows, as well as the user-defined state (see [Working with State]({{ site.baseurl }}/dev/state.html)) consistently to provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured [state backend]({{ site.baseurl }}/dev/state_backends.html).
-
-The [docs on streaming fault tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html) describe in detail the technique behind Flink's streaming fault tolerance mechanism.
-
-To enable checkpointing, call `enableCheckpointing(n)` on the `StreamExecutionEnvironment`, where *n* is the checkpoint interval in milliseconds.
-
-Other parameters for checkpointing include:
-
-- *exactly-once vs. at-least-once*: You can optionally pass a mode to the `enableCheckpointing(n)` method to choose between the two guarantee levels.
- Exactly-once is preferrable for most applications. At-least-once may be relevant for certain super-low-latency (consistently few milliseconds) applications.
-
-- *number of concurrent checkpoints*: By default, the system will not trigger another checkpoint while one is still in progress. This ensures that the topology does not spend too much time on checkpoints and not make progress with processing the streams. It is possible to allow for multiple overlapping checkpoints, which is interesting for pipelines that have a certain processing delay (for example because the functions call external services that need some time to respond) but that still want to do very frequent checkpoints (100s of milliseconds) to re-process very little upon failures.
-
-- *checkpoint timeout*: The time after which a checkpoint-in-progress is aborted, if it did not complete by then.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
-// start a checkpoint every 1000 ms
-env.enableCheckpointing(1000);
-
-// advanced options:
-
-// set mode to exactly-once (this is the default)
-env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
-
-// checkpoints have to complete within one minute, or are discarded
-env.getCheckpointConfig().setCheckpointTimeout(60000);
-
-// allow only one checkpoint to be in progress at the same time
-env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment()
-
-// start a checkpoint every 1000 ms
-env.enableCheckpointing(1000)
-
-// advanced options:
-
-// set mode to exactly-once (this is the default)
-env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
-
-// checkpoints have to complete within one minute, or are discarded
-env.getCheckpointConfig.setCheckpointTimeout(60000)
-
-// allow only one checkpoint to be in progress at the same time
-env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
-{% endhighlight %}
-</div>
-</div>
-
-{% top %}
-
-### Externalized Checkpoints
-
-You can configure periodic checkpoints to be persisted externally. Externalized checkpoints write their meta data out to persistent storage and are *not* automatically cleaned up when the job fails. This way, you will have a checkpoint around to resume from if your job fails.
-
-```java
-CheckpoingConfig config = env.getCheckpointConfig();
-config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
-```
-
-The `ExternalizedCheckpointCleanup` mode configures what happens with externalized checkpoints when you cancel the job:
-
-- **`ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION`**: Retain the externalized checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.
-
-- **`ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION`**: Delete the externalized checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.
-
-The **target directory** for the checkpoint is determined from the default checkpoint directory configuration. This is configured via the configuration key `state.checkpoints.dir`, which should point to the desired target directory:
-
-```
-state.checkpoints.dir: hdfs:///checkpoints/
-```
-
-This directory will then contain the checkpoint meta data required to restore the checkpoint. The actual checkpoint files will still be available in their configured directory. You currently can only set this via the configuration files.
-
-Follow the [savepoint guide]({{ site.baseurl }}/setup/cli.html#savepoints) when you want to resume from a specific checkpoint.
-
-### Fault Tolerance Guarantees of Data Sources and Sinks
-
-Flink can guarantee exactly-once state updates to user-defined state only when the source participates in the
-snapshotting mechanism. The following table lists the state update guarantees of Flink coupled with the bundled connectors.
-
-Please read the documentation of each connector to understand the details of the fault tolerance guarantees.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 25%">Source</th>
- <th class="text-left" style="width: 25%">Guarantees</th>
- <th class="text-left">Notes</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>Apache Kafka</td>
- <td>exactly once</td>
- <td>Use the appropriate Kafka connector for your version</td>
- </tr>
- <tr>
- <td>AWS Kinesis Streams</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>RabbitMQ</td>
- <td>at most once (v 0.10) / exactly once (v 1.0) </td>
- <td></td>
- </tr>
- <tr>
- <td>Twitter Streaming API</td>
- <td>at most once</td>
- <td></td>
- </tr>
- <tr>
- <td>Collections</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>Files</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>Sockets</td>
- <td>at most once</td>
- <td></td>
- </tr>
- </tbody>
-</table>
-
-To guarantee end-to-end exactly-once record delivery (in addition to exactly-once state semantics), the data sink needs
-to take part in the checkpointing mechanism. The following table lists the delivery guarantees (assuming exactly-once
-state updates) of Flink coupled with bundled sinks:
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 25%">Sink</th>
- <th class="text-left" style="width: 25%">Guarantees</th>
- <th class="text-left">Notes</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>HDFS rolling sink</td>
- <td>exactly once</td>
- <td>Implementation depends on Hadoop version</td>
- </tr>
- <tr>
- <td>Elasticsearch</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Kafka producer</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Cassandra sink</td>
- <td>at least once / exactly once</td>
- <td>exactly once only for idempotent updates</td>
- </tr>
- <tr>
- <td>AWS Kinesis Streams</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>File sinks</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Socket sinks</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Standard output</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Redis sink</td>
- <td>at least once</td>
- <td></td>
- </tr>
- </tbody>
-</table>
-
-{% top %}
-
-## Restart Strategies
-
Flink supports different restart strategies which control how the jobs are restarted in case of a failure.
The cluster can be started with a default restart strategy which is always used when no job specific restart strategy has been defined.
In case that the job is submitted with a restart strategy, this strategy overrides the cluster's default setting.
@@ -246,7 +29,6 @@ In case that the job is submitted with a restart strategy, this strategy overrid
The default restart strategy is set via Flink's configuration file `flink-conf.yaml`.
The configuration parameter *restart-strategy* defines which strategy is taken.
Per default, the no-restart strategy is used.
-When checkpointing is activated and no restart strategy has been configured, the job will be restarted infinitely often.
See the following list of available restart strategies to learn what values are supported.
Each restart strategy comes with its own set of parameters which control its behaviour.
@@ -306,7 +88,7 @@ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(
{% top %}
-### Fixed Delay Restart Strategy
+## Fixed Delay Restart Strategy
The fixed delay restart strategy attempts a given number of times to restart the job.
If the maximum number of attempts is exceeded, the job eventually fails.
@@ -329,12 +111,12 @@ restart-strategy: fixed-delay
<tbody>
<tr>
<td><it>restart-strategy.fixed-delay.attempts</it></td>
- <td>The number of times that Flink retries the execution before the job is declared as failed</td>
+ <td>Number of restart attempts</td>
<td>1</td>
</tr>
<tr>
<td><it>restart-strategy.fixed-delay.delay</it></td>
- <td>Delay between two consecutive restart attempts. Delaying the retry means that after a failed execution, the re-execution does not start immediately, but only after a certain delay. Delaying the retries can be helpful when the program interacts with external systems where for example connections or pending transactions should reach a timeout before re-execution is attempted.</td>
+ <td>Delay between two consecutive restart attempts</td>
<td><it>akka.ask.timeout</it></td>
</tr>
</tbody>
@@ -368,9 +150,23 @@ env.setRestartStrategy(RestartStrategies.fixedDelayRestart(
</div>
</div>
+### Restart Attempts
+
+The number of times that Flink retries the execution before the job is declared as failed is configurable via the *restart-strategy.fixed-delay.attempts* parameter.
+
+The default value is **1**.
+
+### Retry Delays
+
+Execution retries can be configured to be delayed. Delaying the retry means that after a failed execution, the re-execution does not start immediately, but only after a certain delay.
+
+Delaying the retries can be helpful when the program interacts with external systems where for example connections or pending transactions should reach a timeout before re-execution is attempted.
+
+The default value is the value of *akka.ask.timeout*.
+
{% top %}
-### Failure Rate Restart Strategy
+## Failure Rate Restart Strategy
The failure rate restart strategy restarts job after failure, but when `failure rate` (failures per time interval) is exceeded, the job eventually fails.
In-between two consecutive restart attempts, the restart strategy waits a fixed amount of time.
@@ -441,7 +237,7 @@ env.setRestartStrategy(RestartStrategies.failureRateRestart(
{% top %}
-### No Restart Strategy
+## No Restart Strategy
The job fails directly and no restart is attempted.
@@ -466,7 +262,7 @@ env.setRestartStrategy(RestartStrategies.noRestart())
</div>
</div>
-### Fallback Restart Strategy
+## Fallback Restart Strategy
The cluster defined restart strategy is used.
This helpful for streaming programs which enable checkpointing.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/flink_on_windows.md
----------------------------------------------------------------------
diff --git a/docs/setup/flink_on_windows.md b/docs/setup/flink_on_windows.md
new file mode 100644
index 0000000..2cbc163
--- /dev/null
+++ b/docs/setup/flink_on_windows.md
@@ -0,0 +1,86 @@
+---
+title: "Running Flink on Windows"
+nav-parent_id: start
+nav-pos: 12
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+If you want to run Flink locally on a Windows machine you need to [download](http://flink.apache.org/downloads.html) and unpack the binary Flink distribution. After that you can either use the **Windows Batch** file (`.bat`), or use **Cygwin** to run the Flink Jobmanager.
+
+## Starting with Windows Batch Files
+
+To start Flink in local mode from the *Windows Batch*, open the command window, navigate to the `bin/` directory of Flink and run `start-local.bat`.
+
+Note: The ``bin`` folder of your Java Runtime Environment must be included in Window's ``%PATH%`` variable. Follow this [guide](http://www.java.com/en/download/help/path.xml) to add Java to the ``%PATH%`` variable.
+
+~~~bash
+$ cd flink
+$ cd bin
+$ start-local.bat
+Starting Flink job manager. Web interface by default on http://localhost:8081/.
+Do not close this batch window. Stop job manager by pressing Ctrl+C.
+~~~
+
+After that, you need to open a second terminal to run jobs using `flink.bat`.
+
+{% top %}
+
+## Starting with Cygwin and Unix Scripts
+
+With *Cygwin* you need to start the Cygwin Terminal, navigate to your Flink directory and run the `start-local.sh` script:
+
+~~~bash
+$ cd flink
+$ bin/start-local.sh
+Starting jobmanager.
+~~~
+
+{% top %}
+
+## Installing Flink from Git
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similiar to this one:
+
+~~~bash
+c:/flink/bin/start-local.sh: line 30: $'\r': command not found
+~~~
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
+
+ ~~~bash
+ cd; pwd
+ ~~~
+
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
+
+~~~bash
+export SHELLOPTS
+set -o igncr
+~~~
+
+Save the file and open a new bash shell.
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/setup/index.md
----------------------------------------------------------------------
diff --git a/docs/setup/index.md b/docs/setup/index.md
index bf34fe0..bc8a9dd 100644
--- a/docs/setup/index.md
+++ b/docs/setup/index.md
@@ -1,9 +1,9 @@
---
-title: "Setup & Operations"
+title: "Deployment & Operations"
nav-id: "setup"
-nav-title: '<i class="fa fa-cogs" aria-hidden="true"></i> Setup & Operations'
+nav-title: '<i class="fa fa-sliders title maindish" aria-hidden="true"></i> Deployment & Operations'
nav-parent_id: root
-nav-pos: 2
+nav-pos: 6
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
[3/4] flink git commit: [FLINK-5258] [docs] Reorganize the docs to
improve navigation and reduce duplication
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/guarantees.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/guarantees.md b/docs/dev/connectors/guarantees.md
new file mode 100644
index 0000000..a75f0e0
--- /dev/null
+++ b/docs/dev/connectors/guarantees.md
@@ -0,0 +1,143 @@
+---
+title: "Fault Tolerance Guarantees of Data Sources and Sinks"
+nav-title: Fault Tolerance Guarantees
+nav-parent_id: connectors
+nav-pos: 0
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Flink's fault tolerance mechanism recovers programs in the presence of failures and
+continues to execute them. Such failures include machine hardware failures, network failures,
+transient program failures, etc.
+
+Flink can guarantee exactly-once state updates to user-defined state only when the source participates in the
+snapshotting mechanism. The following table lists the state update guarantees of Flink coupled with the bundled connectors.
+
+Please read the documentation of each connector to understand the details of the fault tolerance guarantees.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 25%">Source</th>
+ <th class="text-left" style="width: 25%">Guarantees</th>
+ <th class="text-left">Notes</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Apache Kafka</td>
+ <td>exactly once</td>
+ <td>Use the appropriate Kafka connector for your version</td>
+ </tr>
+ <tr>
+ <td>AWS Kinesis Streams</td>
+ <td>exactly once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>RabbitMQ</td>
+ <td>at most once (v 0.10) / exactly once (v 1.0) </td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Twitter Streaming API</td>
+ <td>at most once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Collections</td>
+ <td>exactly once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Files</td>
+ <td>exactly once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Sockets</td>
+ <td>at most once</td>
+ <td></td>
+ </tr>
+ </tbody>
+</table>
+
+To guarantee end-to-end exactly-once record delivery (in addition to exactly-once state semantics), the data sink needs
+to take part in the checkpointing mechanism. The following table lists the delivery guarantees (assuming exactly-once
+state updates) of Flink coupled with bundled sinks:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 25%">Sink</th>
+ <th class="text-left" style="width: 25%">Guarantees</th>
+ <th class="text-left">Notes</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>HDFS rolling sink</td>
+ <td>exactly once</td>
+ <td>Implementation depends on Hadoop version</td>
+ </tr>
+ <tr>
+ <td>Elasticsearch</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Kafka producer</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Cassandra sink</td>
+ <td>at least once / exactly once</td>
+ <td>exactly once only for idempotent updates</td>
+ </tr>
+ <tr>
+ <td>AWS Kinesis Streams</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>File sinks</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Socket sinks</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Standard output</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>Redis sink</td>
+ <td>at least once</td>
+ <td></td>
+ </tr>
+ </tbody>
+</table>
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/index.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/index.md b/docs/dev/connectors/index.md
index 59b5e7b..5de5300 100644
--- a/docs/dev/connectors/index.md
+++ b/docs/dev/connectors/index.md
@@ -2,8 +2,8 @@
title: "Streaming Connectors"
nav-id: connectors
nav-title: Connectors
-nav-parent_id: dev
-nav-pos: 7
+nav-parent_id: streaming
+nav-pos: 30
nav-show_overview: true
---
<!--
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index e3dc821..0798f0b 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -82,7 +82,7 @@ Then, import the connector in your maven project:
</dependency>
{% endhighlight %}
-Note that the streaming connectors are currently not part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+Note that the streaming connectors are currently not part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/linking).
### Installing Apache Kafka
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/kinesis.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kinesis.md b/docs/dev/connectors/kinesis.md
index c54239d..480a97d 100644
--- a/docs/dev/connectors/kinesis.md
+++ b/docs/dev/connectors/kinesis.md
@@ -51,7 +51,7 @@ mvn clean install -Pinclude-kinesis -DskipTests
The streaming connectors are not part of the binary distribution. See how to link with them for cluster
-execution [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+execution [here]({{site.baseurl}}/dev/linking).
### Using the Amazon Kinesis Streams Service
Follow the instructions from the [Amazon Kinesis Streams Developer Guide](https://docs.aws.amazon.com/streams/latest/dev/learning-kinesis-module-one-create-stream.html)
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/nifi.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/nifi.md b/docs/dev/connectors/nifi.md
index 924a80b..bdbd808 100644
--- a/docs/dev/connectors/nifi.md
+++ b/docs/dev/connectors/nifi.md
@@ -37,7 +37,7 @@ following dependency to your project:
Note that the streaming connectors are currently not part of the binary
distribution. See
-[here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution)
+[here]({{site.baseurl}}/dev/linking)
for information about how to package the program with the libraries for
cluster execution.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/rabbitmq.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/rabbitmq.md b/docs/dev/connectors/rabbitmq.md
index 02def40..1b621c0 100644
--- a/docs/dev/connectors/rabbitmq.md
+++ b/docs/dev/connectors/rabbitmq.md
@@ -33,7 +33,7 @@ This connector provides access to data streams from [RabbitMQ](http://www.rabbit
</dependency>
{% endhighlight %}
-Note that the streaming connectors are currently not part of the binary distribution. See linking with them for cluster execution [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+Note that the streaming connectors are currently not part of the binary distribution. See linking with them for cluster execution [here]({{site.baseurl}}/dev/linking).
#### Installing RabbitMQ
Follow the instructions from the [RabbitMQ download page](http://www.rabbitmq.com/download.html). After the installation the server automatically starts, and the application connecting to RabbitMQ can be launched.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/redis.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/redis.md b/docs/dev/connectors/redis.md
index a987b90..0e3287d 100644
--- a/docs/dev/connectors/redis.md
+++ b/docs/dev/connectors/redis.md
@@ -35,7 +35,7 @@ following dependency to your project:
{% endhighlight %}
Version Compatibility: This module is compatible with Redis 2.8.5.
-Note that the streaming connectors are currently not part of the binary distribution. You need to link them for cluster execution [explicitly]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+Note that the streaming connectors are currently not part of the binary distribution. You need to link them for cluster execution [explicitly]({{site.baseurl}}/dev/linking).
#### Installing Redis
Follow the instructions from the [Redis download page](http://redis.io/download).
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/twitter.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/twitter.md b/docs/dev/connectors/twitter.md
index e92e51d..0ccbbff 100644
--- a/docs/dev/connectors/twitter.md
+++ b/docs/dev/connectors/twitter.md
@@ -36,7 +36,7 @@ To use this connector, add the following dependency to your project:
{% endhighlight %}
Note that the streaming connectors are currently not part of the binary distribution.
-See linking with them for cluster execution [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+See linking with them for cluster execution [here]({{site.baseurl}}/dev/linking).
#### Authentication
In order to connect to the Twitter stream the user has to register their program and acquire the necessary information for the authentication. The process is described below.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/custom_serializers.md
----------------------------------------------------------------------
diff --git a/docs/dev/custom_serializers.md b/docs/dev/custom_serializers.md
new file mode 100644
index 0000000..2b72ca0
--- /dev/null
+++ b/docs/dev/custom_serializers.md
@@ -0,0 +1,112 @@
+---
+title: Register a custom serializer for your Flink program
+nav-title: Custom Serializers
+nav-parent_id: types
+nav-pos: 10
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+If you use a custom type in your Flink program which cannot be serialized by the
+Flink type serializer, Flink falls back to using the generic Kryo
+serializer. You may register your own serializer or a serialization system like
+Google Protobuf or Apache Thrift with Kryo. To do that, simply register the type
+class and the serializer in the `ExecutionConfig` of your Flink program.
+
+
+{% highlight java %}
+final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+
+// register the class of the serializer as serializer for a type
+env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, MyCustomSerializer.class);
+
+// register an instance as serializer for a type
+MySerializer mySerializer = new MySerializer();
+env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, mySerializer);
+{% endhighlight %}
+
+Note that your custom serializer has to extend Kryo's Serializer class. In the
+case of Google Protobuf or Apache Thrift, this has already been done for
+you:
+
+{% highlight java %}
+
+final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+
+// register the Google Protobuf serializer with Kryo
+env.getConfig().registerTypeWithKryoSerializer(MyCustomType.class, ProtobufSerializer.class);
+
+// register the serializer included with Apache Thrift as the standard serializer
+// TBaseSerializer states it should be initialized as a default Kryo serializer
+env.getConfig().addDefaultKryoSerializer(MyCustomType.class, TBaseSerializer.class);
+
+{% endhighlight %}
+
+For the above example to work, you need to include the necessary dependencies in
+your Maven project file (pom.xml). In the dependency section, add the following
+for Apache Thrift:
+
+{% highlight xml %}
+
+<dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>chill-thrift</artifactId>
+ <version>0.5.2</version>
+</dependency>
+<!-- libthrift is required by chill-thrift -->
+<dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>0.6.1</version>
+ <exclusions>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ </exclusion>
+ </exclusions>
+</dependency>
+
+{% endhighlight %}
+
+For Google Protobuf you need the following Maven dependency:
+
+{% highlight xml %}
+
+<dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>chill-protobuf</artifactId>
+ <version>0.5.2</version>
+</dependency>
+<!-- We need protobuf for chill-protobuf -->
+<dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ <version>2.5.0</version>
+</dependency>
+
+{% endhighlight %}
+
+
+Please adjust the versions of both libraries as needed.
+
+
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/datastream_api.md
----------------------------------------------------------------------
diff --git a/docs/dev/datastream_api.md b/docs/dev/datastream_api.md
index 1b167ac..85866f7 100644
--- a/docs/dev/datastream_api.md
+++ b/docs/dev/datastream_api.md
@@ -1,8 +1,10 @@
---
title: "Flink DataStream API Programming Guide"
nav-title: Streaming (DataStream API)
-nav-parent_id: apis
-nav-pos: 2
+nav-id: streaming
+nav-parent_id: dev
+nav-show_overview: true
+nav-pos: 10
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -208,7 +210,7 @@ dataStream.filter(new FilterFunction<Integer>() {
<td><strong>KeyBy</strong><br>DataStream → KeyedStream</td>
<td>
<p>Logically partitions a stream into disjoint partitions, each partition containing elements of the same key.
- Internally, this is implemented with hash partitioning. See <a href="#specifying-keys">keys</a> on how to specify keys.
+ Internally, this is implemented with hash partitioning. See <a href="/dev/api_concepts#specifying-keys">keys</a> on how to specify keys.
This transformation returns a KeyedDataStream.</p>
{% highlight java %}
dataStream.keyBy("someKey") // Key by field "someKey"
@@ -595,7 +597,7 @@ dataStream.filter { _ != 0 }
<td><strong>KeyBy</strong><br>DataStream → KeyedStream</td>
<td>
<p>Logically partitions a stream into disjoint partitions, each partition containing elements of the same key.
- Internally, this is implemented with hash partitioning. See <a href="#specifying-keys">keys</a> on how to specify keys.
+ Internally, this is implemented with hash partitioning. See <a href="/dev/api_concepts#specifying-keys">keys</a> on how to specify keys.
This transformation returns a KeyedDataStream.</p>
{% highlight scala %}
dataStream.keyBy("someKey") // Key by field "someKey"
@@ -1408,8 +1410,8 @@ Collection-based:
Custom:
-- `addSource` - Attache a new source function. For example, to read from Apache Kafka you can use
- `addSource(new FlinkKafkaConsumer08<>(...))`. See [connectors]({{ site.baseurl }}/apis/streaming/connectors/) for more details.
+- `addSource` - Attach a new source function. For example, to read from Apache Kafka you can use
+ `addSource(new FlinkKafkaConsumer08<>(...))`. See [connectors]({{ site.baseurl }}/dev/connectors/) for more details.
</div>
</div>
@@ -1608,7 +1610,7 @@ Execution Parameters
The `StreamExecutionEnvironment` contains the `ExecutionConfig` which allows to set job specific configuration values for the runtime.
-Please refer to [execution configuration]({{ site.baseurl }}/dev/api_concepts.html#execution-configuration)
+Please refer to [execution configuration]({{ site.baseurl }}/dev/execution_configuration)
for an explanation of most parameters. These parameters pertain specifically to the DataStream API:
- `enableTimestamps()` / **`disableTimestamps()`**: Attach a timestamp to each event emitted from a source.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/event_time.md
----------------------------------------------------------------------
diff --git a/docs/dev/event_time.md b/docs/dev/event_time.md
index 7375a0f..5ab5feb 100644
--- a/docs/dev/event_time.md
+++ b/docs/dev/event_time.md
@@ -2,8 +2,8 @@
title: "Event Time"
nav-id: event_time
nav-show_overview: true
-nav-parent_id: dev
-nav-pos: 4
+nav-parent_id: streaming
+nav-pos: 20
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/execution.md
----------------------------------------------------------------------
diff --git a/docs/dev/execution.md b/docs/dev/execution.md
new file mode 100644
index 0000000..4f613e0
--- /dev/null
+++ b/docs/dev/execution.md
@@ -0,0 +1,24 @@
+---
+title: "Managing Execution"
+nav-id: execution
+nav-parent_id: dev
+nav-pos: 60
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/execution_configuration.md
----------------------------------------------------------------------
diff --git a/docs/dev/execution_configuration.md b/docs/dev/execution_configuration.md
new file mode 100644
index 0000000..1f66058
--- /dev/null
+++ b/docs/dev/execution_configuration.md
@@ -0,0 +1,86 @@
+---
+title: "Execution Configuration"
+nav-parent_id: execution
+nav-pos: 10
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The `StreamExecutionEnvironment` contains the `ExecutionConfig` which allows to set job specific configuration values for the runtime.
+To change the defaults that affect all jobs, see [Configuration]({{ site.baseurl }}/setup/config).
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ExecutionConfig executionConfig = env.getConfig();
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+var executionConfig = env.getConfig
+{% endhighlight %}
+</div>
+</div>
+
+The following configuration options are available: (the default is bold)
+
+- **`enableClosureCleaner()`** / `disableClosureCleaner()`. The closure cleaner is enabled by default. The closure cleaner removes unneeded references to the surrounding class of anonymous functions inside Flink programs.
+With the closure cleaner disabled, it might happen that an anonymous user function is referencing the surrounding class, which is usually not Serializable. This will lead to exceptions by the serializer.
+
+- `getParallelism()` / `setParallelism(int parallelism)` Set the default parallelism for the job.
+
+- `getMaxParallelism()` / `setMaxParallelism(int parallelism)` Set the default maximum parallelism for the job. This setting determines the maximum degree of parallelism and specifies the upper limit for dynamic scaling.
+
+- `getNumberOfExecutionRetries()` / `setNumberOfExecutionRetries(int numberOfExecutionRetries)` Sets the number of times that failed tasks are re-executed. A value of zero effectively disables fault tolerance. A value of `-1` indicates that the system default value (as defined in the configuration) should be used.
+
+- `getExecutionRetryDelay()` / `setExecutionRetryDelay(long executionRetryDelay)` Sets the delay in milliseconds that the system waits after a job has failed, before re-executing it. The delay starts after all tasks have been successfully been stopped on the TaskManagers, and once the delay is past, the tasks are re-started. This parameter is useful to delay re-execution in order to let certain time-out related failures surface fully (like broken connections that have not fully timed out), before attempting a re-execution and immediately failing again due to the same problem. This parameter only has an effect if the number of execution re-tries is one or more.
+
+- `getExecutionMode()` / `setExecutionMode()`. The default execution mode is PIPELINED. Sets the execution mode to execute the program. The execution mode defines whether data exchanges are performed in a batch or on a pipelined manner.
+
+- `enableForceKryo()` / **`disableForceKryo`**. Kryo is not forced by default. Forces the GenericTypeInformation to use the Kryo serializer for POJOS even though we could analyze them as a POJO. In some cases this might be preferable. For example, when Flink's internal serializers fail to handle a POJO properly.
+
+- `enableForceAvro()` / **`disableForceAvro()`**. Avro is not forced by default. Forces the Flink AvroTypeInformation to use the Avro serializer instead of Kryo for serializing Avro POJOs.
+
+- `enableObjectReuse()` / **`disableObjectReuse()`** By default, objects are not reused in Flink. Enabling the object reuse mode will instruct the runtime to reuse user objects for better performance. Keep in mind that this can lead to bugs when the user-code function of an operation is not aware of this behavior.
+
+- **`enableSysoutLogging()`** / `disableSysoutLogging()` JobManager status updates are printed to `System.out` by default. This setting allows to disable this behavior.
+
+- `getGlobalJobParameters()` / `setGlobalJobParameters()` This method allows users to set custom objects as a global configuration for the job. Since the `ExecutionConfig` is accessible in all user defined functions, this is an easy method for making configuration globally available in a job.
+
+- `addDefaultKryoSerializer(Class<?> type, Serializer<?> serializer)` Register a Kryo serializer instance for the given `type`.
+
+- `addDefaultKryoSerializer(Class<?> type, Class<? extends Serializer<?>> serializerClass)` Register a Kryo serializer class for the given `type`.
+
+- `registerTypeWithKryoSerializer(Class<?> type, Serializer<?> serializer)` Register the given type with Kryo and specify a serializer for it. By registering a type with Kryo, the serialization of the type will be much more efficient.
+
+- `registerKryoType(Class<?> type)` If the type ends up being serialized with Kryo, then it will be registered at Kryo to make sure that only tags (integer IDs) are written. If a type is not registered with Kryo, its entire class-name will be serialized with every instance, leading to much higher I/O costs.
+
+- `registerPojoType(Class<?> type)` Registers the given type with the serialization stack. If the type is eventually serialized as a POJO, then the type is registered with the POJO serializer. If the type ends up being serialized with Kryo, then it will be registered at Kryo to make sure that only tags are written. If a type is not registered with Kryo, its entire class-name will be serialized with every instance, leading to much higher I/O costs.
+
+Note that types registered with `registerKryoType()` are not available to Flink's Kryo serializer instance.
+
+- `disableAutoTypeRegistration()` Automatic type registration is enabled by default. The automatic type registration is registering all types (including sub-types) used by usercode with Kryo and the POJO serializer.
+
+- `setTaskCancellationInterval(long interval)` Sets the the interval (in milliseconds) to wait between consecutive attempts to cancel a running task. When a task is canceled a new thread is created which periodically calls `interrupt()` on the task thread, if the task thread does not terminate within a certain time. This parameter refers to the time between consecutive calls to `interrupt()` and is set by default to **30000** milliseconds, or **30 seconds**.
+
+The `RuntimeContext` which is accessible in `Rich*` functions through the `getRuntimeContext()` method also allows to access the `ExecutionConfig` in all user defined functions.
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/execution_plans.md
----------------------------------------------------------------------
diff --git a/docs/dev/execution_plans.md b/docs/dev/execution_plans.md
new file mode 100644
index 0000000..881c54e
--- /dev/null
+++ b/docs/dev/execution_plans.md
@@ -0,0 +1,80 @@
+---
+title: "Execution Plans"
+nav-parent_id: execution
+nav-pos: 40
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Depending on various parameters such as data size or number of machines in the cluster, Flink's
+optimizer automatically chooses an execution strategy for your program. In many cases, it can be
+useful to know how exactly Flink will execute your program.
+
+__Plan Visualization Tool__
+
+Flink comes packaged with a visualization tool for execution plans. The HTML document containing
+the visualizer is located under ```tools/planVisualizer.html```. It takes a JSON representation of
+the job execution plan and visualizes it as a graph with complete annotations of execution
+strategies.
+
+The following code shows how to print the execution plan JSON from your program:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+
+...
+
+System.out.println(env.getExecutionPlan());
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = ExecutionEnvironment.getExecutionEnvironment
+
+...
+
+println(env.getExecutionPlan())
+{% endhighlight %}
+</div>
+</div>
+
+
+To visualize the execution plan, do the following:
+
+1. **Open** ```planVisualizer.html``` with your web browser,
+2. **Paste** the JSON string into the text field, and
+3. **Press** the draw button.
+
+After these steps, a detailed execution plan will be visualized.
+
+<img alt="A flink job execution graph." src="{{ site.baseurl }}/fig/plan_visualizer.png" width="80%">
+
+
+__Web Interface__
+
+Flink offers a web interface for submitting and executing jobs. The interface is part of the JobManager's
+web interface for monitoring, per default running on port 8081. Job submission via this interfaces requires
+that you have set `jobmanager.web.submit.enable: true` in `flink-conf.yaml`.
+
+You may specify program arguments before the job is executed. The plan visualization enables you to show
+the execution plan before executing the Flink job.
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/index.md
----------------------------------------------------------------------
diff --git a/docs/dev/index.md b/docs/dev/index.md
index 67916c1..8b96672 100644
--- a/docs/dev/index.md
+++ b/docs/dev/index.md
@@ -1,9 +1,9 @@
---
title: "Application Development"
nav-id: dev
-nav-title: '<i class="fa fa-code" aria-hidden="true"></i> Application Development'
+nav-title: '<i class="fa fa-code title maindish" aria-hidden="true"></i> Application Development'
nav-parent_id: root
-nav-pos: 3
+nav-pos: 5
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/java8.md
----------------------------------------------------------------------
diff --git a/docs/dev/java8.md b/docs/dev/java8.md
index 3792e27..e98f748 100644
--- a/docs/dev/java8.md
+++ b/docs/dev/java8.md
@@ -1,7 +1,7 @@
---
title: "Java 8"
-nav-parent_id: apis
-nav-pos: 105
+nav-parent_id: api-concepts
+nav-pos: 20
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/libraries.md
----------------------------------------------------------------------
diff --git a/docs/dev/libraries.md b/docs/dev/libraries.md
index dc22e97..586637b 100644
--- a/docs/dev/libraries.md
+++ b/docs/dev/libraries.md
@@ -2,7 +2,7 @@
title: "Libraries"
nav-id: libs
nav-parent_id: dev
-nav-pos: 8
+nav-pos: 80
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/libs/cep.md
----------------------------------------------------------------------
diff --git a/docs/dev/libs/cep.md b/docs/dev/libs/cep.md
index d27cf9f..c30d37b 100644
--- a/docs/dev/libs/cep.md
+++ b/docs/dev/libs/cep.md
@@ -37,7 +37,7 @@ because these are used for comparing and matching events.
## Getting Started
-If you want to jump right in, you have to [set up a Flink program]({{ site.baseurl }}/dev/api_concepts.html#linking-with-flink).
+If you want to jump right in, you have to [set up a Flink program]({{ site.baseurl }}/dev/linking_with_flink).
Next, you have to add the FlinkCEP dependency to the `pom.xml` of your project.
<div class="codetabs" markdown="1">
@@ -63,7 +63,7 @@ Next, you have to add the FlinkCEP dependency to the `pom.xml` of your project.
</div>
Note that FlinkCEP is currently not part of the binary distribution.
-See linking with it for cluster execution [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+See linking with it for cluster execution [here]({{site.baseurl}}/dev/linking).
Now you can start writing your first CEP program using the pattern API.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/libs/gelly/index.md
----------------------------------------------------------------------
diff --git a/docs/dev/libs/gelly/index.md b/docs/dev/libs/gelly/index.md
index db7073f..0877e2f 100644
--- a/docs/dev/libs/gelly/index.md
+++ b/docs/dev/libs/gelly/index.md
@@ -62,7 +62,7 @@ Add the following dependency to your `pom.xml` to use Gelly.
</div>
</div>
-Note that Gelly is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+Note that Gelly is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/linking).
The remaining sections provide a description of available methods and present several examples of how to use Gelly and how to mix it with the Flink DataSet API.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/libs/ml/index.md
----------------------------------------------------------------------
diff --git a/docs/dev/libs/ml/index.md b/docs/dev/libs/ml/index.md
index d01e18e..dcd3e0a 100644
--- a/docs/dev/libs/ml/index.md
+++ b/docs/dev/libs/ml/index.md
@@ -68,7 +68,7 @@ FlinkML currently supports the following algorithms:
You can check out our [quickstart guide](quickstart.html) for a comprehensive getting started
example.
-If you want to jump right in, you have to [set up a Flink program]({{ site.baseurl }}/dev/api_concepts.html#linking-with-flink).
+If you want to jump right in, you have to [set up a Flink program]({{ site.baseurl }}/dev/linking_with_flink).
Next, you have to add the FlinkML dependency to the `pom.xml` of your project.
{% highlight xml %}
@@ -80,7 +80,7 @@ Next, you have to add the FlinkML dependency to the `pom.xml` of your project.
{% endhighlight %}
Note that FlinkML is currently not part of the binary distribution.
-See linking with it for cluster execution [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+See linking with it for cluster execution [here]({{site.baseurl}}/dev/linking).
Now you can start solving your analysis task.
The following code snippet shows how easy it is to train a multiple linear regression model.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/libs/ml/quickstart.md
----------------------------------------------------------------------
diff --git a/docs/dev/libs/ml/quickstart.md b/docs/dev/libs/ml/quickstart.md
index 7ba3ed5..29f2fec 100644
--- a/docs/dev/libs/ml/quickstart.md
+++ b/docs/dev/libs/ml/quickstart.md
@@ -55,7 +55,7 @@ through [principal components analysis](https://en.wikipedia.org/wiki/Principal_
## Linking with FlinkML
In order to use FlinkML in your project, first you have to
-[set up a Flink program]({{ site.baseurl }}/dev/api_concepts.html#linking-with-flink).
+[set up a Flink program]({{ site.baseurl }}/dev/linking_with_flink).
Next, you have to add the FlinkML dependency to the `pom.xml` of your project:
{% highlight xml %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/linking.md
----------------------------------------------------------------------
diff --git a/docs/dev/linking.md b/docs/dev/linking.md
new file mode 100644
index 0000000..0592617
--- /dev/null
+++ b/docs/dev/linking.md
@@ -0,0 +1,94 @@
+---
+nav-title: "Linking with Optional Modules"
+title: "Linking with modules not contained in the binary distribution"
+nav-parent_id: start
+nav-pos: 10
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The binary distribution contains jar packages in the `lib` folder that are automatically
+provided to the classpath of your distributed programs. Almost all of Flink classes are
+located there with a few exceptions, for example the streaming connectors and some freshly
+added modules. To run code depending on these modules you need to make them accessible
+during runtime, for which we suggest two options:
+
+1. Either copy the required jar files to the `lib` folder onto all of your TaskManagers.
+Note that you have to restart your TaskManagers after this.
+2. Or package them with your code.
+
+The latter version is recommended as it respects the classloader management in Flink.
+
+### Packaging dependencies with your usercode with Maven
+
+To provide these dependencies not included by Flink we suggest two options with Maven.
+
+1. The maven assembly plugin builds a so-called uber-jar (executable jar) containing all your dependencies.
+The assembly configuration is straight-forward, but the resulting jar might become bulky.
+See [maven-assembly-plugin](http://maven.apache.org/plugins/maven-assembly-plugin/usage.html) for further information.
+2. The maven unpack plugin unpacks the relevant parts of the dependencies and
+then packages it with your code.
+
+Using the latter approach in order to bundle the Kafka connector, `flink-connector-kafka`
+you would need to add the classes from both the connector and the Kafka API itself. Add
+the following to your plugins section.
+
+~~~xml
+<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <version>2.9</version>
+ <executions>
+ <execution>
+ <id>unpack</id>
+ <!-- executed just before the package phase -->
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>unpack</goal>
+ </goals>
+ <configuration>
+ <artifactItems>
+ <!-- For Flink connector classes -->
+ <artifactItem>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-kafka</artifactId>
+ <version>{{ site.version }}</version>
+ <type>jar</type>
+ <overWrite>false</overWrite>
+ <outputDirectory>${project.build.directory}/classes</outputDirectory>
+ <includes>org/apache/flink/**</includes>
+ </artifactItem>
+ <!-- For Kafka API classes -->
+ <artifactItem>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka_<YOUR_SCALA_VERSION></artifactId>
+ <version><YOUR_KAFKA_VERSION></version>
+ <type>jar</type>
+ <overWrite>false</overWrite>
+ <outputDirectory>${project.build.directory}/classes</outputDirectory>
+ <includes>kafka/**</includes>
+ </artifactItem>
+ </artifactItems>
+ </configuration>
+ </execution>
+ </executions>
+</plugin>
+~~~
+
+Now when running `mvn clean package` the produced jar includes the required dependencies.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/linking_with_flink.md
----------------------------------------------------------------------
diff --git a/docs/dev/linking_with_flink.md b/docs/dev/linking_with_flink.md
new file mode 100644
index 0000000..73ca677
--- /dev/null
+++ b/docs/dev/linking_with_flink.md
@@ -0,0 +1,146 @@
+---
+title: "Linking with Flink"
+nav-parent_id: start
+nav-pos: 2
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+To write programs with Flink, you need to include the Flink library corresponding to
+your programming language in your project.
+
+The simplest way to do this is to use one of the quickstart scripts: either for
+[Java]({{ site.baseurl }}/quickstart/java_api_quickstart.html) or for [Scala]({{ site.baseurl }}/quickstart/scala_api_quickstart.html). They
+create a blank project from a template (a Maven Archetype), which sets up everything for you. To
+manually create the project, you can use the archetype and create a project by calling:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight bash %}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.flink \
+ -DarchetypeArtifactId=flink-quickstart-java \
+ -DarchetypeVersion={{site.version }}
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight bash %}
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.flink \
+ -DarchetypeArtifactId=flink-quickstart-scala \
+ -DarchetypeVersion={{site.version }}
+{% endhighlight %}
+</div>
+</div>
+
+The archetypes are working for stable releases and preview versions (`-SNAPSHOT`).
+
+If you want to add Flink to an existing Maven project, add the following entry to your
+*dependencies* section in the *pom.xml* file of your project:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight xml %}
+<!-- Use this dependency if you are using the DataStream API -->
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+<!-- Use this dependency if you are using the DataSet API -->
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-java</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-clients{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight xml %}
+<!-- Use this dependency if you are using the DataStream API -->
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+<!-- Use this dependency if you are using the DataSet API -->
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-clients{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+**Important:** When working with the Scala API you must have one of these two imports:
+{% highlight scala %}
+import org.apache.flink.api.scala._
+{% endhighlight %}
+
+or
+
+{% highlight scala %}
+import org.apache.flink.api.scala.createTypeInformation
+{% endhighlight %}
+
+The reason is that Flink analyzes the types that are used in a program and generates serializers
+and comparaters for them. By having either of those imports you enable an implicit conversion
+that creates the type information for Flink operations.
+
+If you would rather use SBT, see [here]({{ site.baseurl }}/quickstart/scala_api_quickstart.html#sbt).
+</div>
+</div>
+
+#### Scala Dependency Versions
+
+Because Scala 2.10 binary is not compatible with Scala 2.11 binary, we provide multiple artifacts
+to support both Scala versions.
+
+Starting from the 0.10 line, we cross-build all Flink modules for both 2.10 and 2.11. If you want
+to run your program on Flink with Scala 2.11, you need to add a `_2.11` suffix to the `artifactId`
+values of the Flink modules in your dependencies section.
+
+If you are looking for building Flink with Scala 2.11, please check
+[build guide]({{ site.baseurl }}/setup/building.html#scala-versions).
+
+#### Hadoop Dependency Versions
+
+If you are using Flink together with Hadoop, the version of the dependency may vary depending on the
+version of Hadoop (or more specifically, HDFS) that you want to use Flink with. Please refer to the
+[downloads page](http://flink.apache.org/downloads.html) for a list of available versions, and instructions
+on how to link with custom versions of Hadoop.
+
+In order to link against the latest SNAPSHOT versions of the code, please follow
+[this guide](http://flink.apache.org/how-to-contribute.html#snapshots-nightly-builds).
+
+The *flink-clients* dependency is only necessary to invoke the Flink program locally (for example to
+run it standalone for testing and debugging). If you intend to only export the program as a JAR
+file and [run it on a cluster]({{ site.baseurl }}/dev/cluster_execution.html), you can skip that dependency.
+
+{% top %}
+
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/local_execution.md
----------------------------------------------------------------------
diff --git a/docs/dev/local_execution.md b/docs/dev/local_execution.md
index a348951..45a39e3 100644
--- a/docs/dev/local_execution.md
+++ b/docs/dev/local_execution.md
@@ -1,7 +1,7 @@
---
title: "Local Execution"
-nav-parent_id: dev
-nav-pos: 11
+nav-parent_id: batch
+nav-pos: 8
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/packaging.md
----------------------------------------------------------------------
diff --git a/docs/dev/packaging.md b/docs/dev/packaging.md
new file mode 100644
index 0000000..ee351ae
--- /dev/null
+++ b/docs/dev/packaging.md
@@ -0,0 +1,77 @@
+---
+title: "Program Packaging and Distributed Execution"
+nav-title: Program Packaging
+nav-parent_id: execution
+nav-pos: 20
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+As described earlier, Flink programs can be executed on
+clusters by using a `remote environment`. Alternatively, programs can be packaged into JAR Files
+(Java Archives) for execution. Packaging the program is a prerequisite to executing them through the
+[command line interface]({{ site.baseurl }}/setup/cli.html).
+
+### Packaging Programs
+
+To support execution from a packaged JAR file via the command line or web interface, a program must
+use the environment obtained by `StreamExecutionEnvironment.getExecutionEnvironment()`. This environment
+will act as the cluster's environment when the JAR is submitted to the command line or web
+interface. If the Flink program is invoked differently than through these interfaces, the
+environment will act like a local environment.
+
+To package the program, simply export all involved classes as a JAR file. The JAR file's manifest
+must point to the class that contains the program's *entry point* (the class with the public
+`main` method). The simplest way to do this is by putting the *main-class* entry into the
+manifest (such as `main-class: org.apache.flinkexample.MyProgram`). The *main-class* attribute is
+the same one that is used by the Java Virtual Machine to find the main method when executing a JAR
+files through the command `java -jar pathToTheJarFile`. Most IDEs offer to include that attribute
+automatically when exporting JAR files.
+
+
+### Packaging Programs through Plans
+
+Additionally, we support packaging programs as *Plans*. Instead of defining a progam in the main
+method and calling
+`execute()` on the environment, plan packaging returns the *Program Plan*, which is a description of
+the program's data flow. To do that, the program must implement the
+`org.apache.flink.api.common.Program` interface, defining the `getPlan(String...)` method. The
+strings passed to that method are the command line arguments. The program's plan can be created from
+the environment via the `ExecutionEnvironment#createProgramPlan()` method. When packaging the
+program's plan, the JAR manifest must point to the class implementing the
+`org.apache.flinkapi.common.Program` interface, instead of the class with the main method.
+
+
+### Summary
+
+The overall procedure to invoke a packaged program is as follows:
+
+1. The JAR's manifest is searched for a *main-class* or *program-class* attribute. If both
+attributes are found, the *program-class* attribute takes precedence over the *main-class*
+attribute. Both the command line and the web interface support a parameter to pass the entry point
+class name manually for cases where the JAR manifest contains neither attribute.
+
+2. If the entry point class implements the `org.apache.flinkapi.common.Program`, then the system
+calls the `getPlan(String...)` method to obtain the program plan to execute.
+
+3. If the entry point class does not implement the `org.apache.flinkapi.common.Program` interface,
+the system will invoke the main method of the class.
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/parallel.md
----------------------------------------------------------------------
diff --git a/docs/dev/parallel.md b/docs/dev/parallel.md
new file mode 100644
index 0000000..8d38884
--- /dev/null
+++ b/docs/dev/parallel.md
@@ -0,0 +1,175 @@
+---
+title: "Parallel Execution"
+nav-parent_id: execution
+nav-pos: 30
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This section describes how the parallel execution of programs can be configured in Flink. A Flink
+program consists of multiple tasks (transformations/operators, data sources, and sinks). A task is split into
+several parallel instances for execution and each parallel instance processes a subset of the task's
+input data. The number of parallel instances of a task is called its *parallelism*.
+
+
+The parallelism of a task can be specified in Flink on different levels.
+
+## Operator Level
+
+The parallelism of an individual operator, data source, or data sink can be defined by calling its
+`setParallelism()` method. For example, like this:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+DataStream<String> text = [...]
+DataStream<Tuple2<String, Integer>> wordCounts = text
+ .flatMap(new LineSplitter())
+ .keyBy(0)
+ .timeWindow(Time.seconds(5))
+ .sum(1).setParallelism(5);
+
+wordCounts.print();
+
+env.execute("Word Count Example");
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+
+val text = [...]
+val wordCounts = text
+ .flatMap{ _.split(" ") map { (_, 1) } }
+ .keyBy(0)
+ .timeWindow(Time.seconds(5))
+ .sum(1).setParallelism(5)
+wordCounts.print()
+
+env.execute("Word Count Example")
+{% endhighlight %}
+</div>
+</div>
+
+## Execution Environment Level
+
+As mentioned [here](#anatomy-of-a-flink-program) Flink programs are executed in the context
+of an execution environment. An
+execution environment defines a default parallelism for all operators, data sources, and data sinks
+it executes. Execution environment parallelism can be overwritten by explicitly configuring the
+parallelism of an operator.
+
+The default parallelism of an execution environment can be specified by calling the
+`setParallelism()` method. To execute all operators, data sources, and data sinks with a parallelism
+of `3`, set the default parallelism of the execution environment as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+env.setParallelism(3);
+
+DataStream<String> text = [...]
+DataStream<Tuple2<String, Integer>> wordCounts = [...]
+wordCounts.print();
+
+env.execute("Word Count Example");
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+env.setParallelism(3)
+
+val text = [...]
+val wordCounts = text
+ .flatMap{ _.split(" ") map { (_, 1) } }
+ .keyBy(0)
+ .timeWindow(Time.seconds(5))
+ .sum(1)
+wordCounts.print()
+
+env.execute("Word Count Example")
+{% endhighlight %}
+</div>
+</div>
+
+## Client Level
+
+The parallelism can be set at the Client when submitting jobs to Flink. The
+Client can either be a Java or a Scala program. One example of such a Client is
+Flink's Command-line Interface (CLI).
+
+For the CLI client, the parallelism parameter can be specified with `-p`. For
+example:
+
+ ./bin/flink run -p 10 ../examples/*WordCount-java*.jar
+
+
+In a Java/Scala program, the parallelism is set as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+
+try {
+ PackagedProgram program = new PackagedProgram(file, args);
+ InetSocketAddress jobManagerAddress = RemoteExecutor.getInetFromHostport("localhost:6123");
+ Configuration config = new Configuration();
+
+ Client client = new Client(jobManagerAddress, config, program.getUserCodeClassLoader());
+
+ // set the parallelism to 10 here
+ client.run(program, 10, true);
+
+} catch (ProgramInvocationException e) {
+ e.printStackTrace();
+}
+
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+try {
+ PackagedProgram program = new PackagedProgram(file, args)
+ InetSocketAddress jobManagerAddress = RemoteExecutor.getInetFromHostport("localhost:6123")
+ Configuration config = new Configuration()
+
+ Client client = new Client(jobManagerAddress, new Configuration(), program.getUserCodeClassLoader())
+
+ // set the parallelism to 10 here
+ client.run(program, 10, true)
+
+} catch {
+ case e: Exception => e.printStackTrace
+}
+{% endhighlight %}
+</div>
+</div>
+
+
+## System Level
+
+A system-wide default parallelism for all execution environments can be defined by setting the
+`parallelism.default` property in `./conf/flink-conf.yaml`. See the
+[Configuration]({{ site.baseurl }}/setup/config.html) documentation for details.
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/quickstarts.md
----------------------------------------------------------------------
diff --git a/docs/dev/quickstarts.md b/docs/dev/quickstarts.md
deleted file mode 100644
index ef21ca6..0000000
--- a/docs/dev/quickstarts.md
+++ /dev/null
@@ -1,24 +0,0 @@
----
-title: "Quickstarts"
-nav-id: quickstarts
-nav-parent_id: dev
-nav-pos: 1
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/scala_api_extensions.md
----------------------------------------------------------------------
diff --git a/docs/dev/scala_api_extensions.md b/docs/dev/scala_api_extensions.md
index ffa6145..0e54ef1 100644
--- a/docs/dev/scala_api_extensions.md
+++ b/docs/dev/scala_api_extensions.md
@@ -1,7 +1,7 @@
---
title: "Scala API Extensions"
-nav-parent_id: apis
-nav-pos: 104
+nav-parent_id: api-concepts
+nav-pos: 10
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/scala_shell.md
----------------------------------------------------------------------
diff --git a/docs/dev/scala_shell.md b/docs/dev/scala_shell.md
index 0728812..a8d1b74 100644
--- a/docs/dev/scala_shell.md
+++ b/docs/dev/scala_shell.md
@@ -1,7 +1,7 @@
---
-title: "Scala Shell"
-nav-parent_id: dev
-nav-pos: 10
+title: "Scala REPL"
+nav-parent_id: start
+nav-pos: 5
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/state.md
----------------------------------------------------------------------
diff --git a/docs/dev/state.md b/docs/dev/state.md
index 37de0a8..6ed20ae 100644
--- a/docs/dev/state.md
+++ b/docs/dev/state.md
@@ -1,7 +1,8 @@
---
-title: "Working with State"
-nav-parent_id: dev
-nav-pos: 3
+title: "State & Checkpointing"
+nav-parent_id: streaming
+nav-id: state
+nav-pos: 40
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -45,6 +46,73 @@ about the available state backends and how to configure them.
* ToC
{:toc}
+Enabling Checkpointing
+-------------------------
+
+Flink has a checkpointing mechanism that recovers streaming jobs after failures. The checkpointing mechanism requires a *persistent* (or *durable*) source that
+can be asked for prior records again (Apache Kafka is a good example of such a source).
+
+The checkpointing mechanism stores the progress in the data sources and data sinks, the state of windows, as well as the user-defined state (see [Working with State]({{ site.baseurl }}/dev/state.html)) consistently to provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends on the configured [state backend]({{ site.baseurl }}/dev/state_backends.html).
+
+The [docs on streaming fault tolerance]({{ site.baseurl }}/internals/stream_checkpointing.html) describe in detail the technique behind Flink's streaming fault tolerance mechanism.
+
+By default, checkpointing is disabled. To enable checkpointing, call `enableCheckpointing(n)` on the `StreamExecutionEnvironment`, where *n* is the checkpoint interval in milliseconds.
+
+Other parameters for checkpointing include:
+
+- *Number of retries*: The `setNumberOfExecutionRerties()` method defines how many times the job is restarted after a failure.
+ When checkpointing is activated, but this value is not explicitly set, the job is restarted infinitely often.
+
+- *exactly-once vs. at-least-once*: You can optionally pass a mode to the `enableCheckpointing(n)` method to choose between the two guarantee levels.
+ Exactly-once is preferrable for most applications. At-least-once may be relevant for certain super-low-latency (consistently few milliseconds) applications.
+
+- *number of concurrent checkpoints*: By default, the system will not trigger another checkpoint while one is still in progress. This ensures that the topology does not spend too much time on checkpoints and not make progress with processing the streams. It is possible to allow for multiple overlapping checkpoints, which is interesting for pipelines that have a certain processing delay (for example because the functions call external services that need some time to respond) but that still want to do very frequent checkpoints (100s of milliseconds) to re-process very little upon failures.
+
+- *checkpoint timeout*: The time after which a checkpoint-in-progress is aborted, if it did not complete by then.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+// start a checkpoint every 1000 ms
+env.enableCheckpointing(1000);
+
+// advanced options:
+
+// set mode to exactly-once (this is the default)
+env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
+
+// checkpoints have to complete within one minute, or are discarded
+env.getCheckpointConfig().setCheckpointTimeout(60000);
+
+// allow only one checkpoint to be in progress at the same time
+env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment()
+
+// start a checkpoint every 1000 ms
+env.enableCheckpointing(1000)
+
+// advanced options:
+
+// set mode to exactly-once (this is the default)
+env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
+
+// checkpoints have to complete within one minute, or are discarded
+env.getCheckpointConfig.setCheckpointTimeout(60000)
+
+// allow only one checkpoint to be in progress at the same time
+env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
## Using the Key/Value State Interface
The Key/Value state interface provides access to different types of state that are all scoped to
@@ -84,7 +152,7 @@ want to retrieve, you create either a `ValueStateDescriptor`, a `ListStateDescri
a `ReducingStateDescriptor`.
State is accessed using the `RuntimeContext`, so it is only possible in *rich functions*.
-Please see [here]({{ site.baseurl }}/apis/common/#specifying-transformation-functions) for
+Please see [here]({{ site.baseurl }}/dev/api_concepts#rich-functions) for
information about that, but we will also see an example shortly. The `RuntimeContext` that
is available in a `RichFunction` has these methods for accessing state:
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/dev/state_backends.md b/docs/dev/state_backends.md
index 31ebb6f..af9934d 100644
--- a/docs/dev/state_backends.md
+++ b/docs/dev/state_backends.md
@@ -1,7 +1,7 @@
---
title: "State Backends"
-nav-parent_id: dev
-nav-pos: 5
+nav-parent_id: setup
+nav-pos: 11
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/table_api.md
----------------------------------------------------------------------
diff --git a/docs/dev/table_api.md b/docs/dev/table_api.md
index 9271803..6ffc23e 100644
--- a/docs/dev/table_api.md
+++ b/docs/dev/table_api.md
@@ -1,8 +1,8 @@
---
title: "Table and SQL"
is_beta: true
-nav-parent_id: apis
-nav-pos: 3
+nav-parent_id: libs
+nav-pos: 0
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -47,7 +47,7 @@ The following dependency must be added to your project in order to use the Table
</dependency>
{% endhighlight %}
-*Note: The Table API is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).*
+*Note: The Table API is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/linking.html).*
Registering Tables
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/types_serialization.md
----------------------------------------------------------------------
diff --git a/docs/dev/types_serialization.md b/docs/dev/types_serialization.md
index 4b8e25f..ea02df0 100644
--- a/docs/dev/types_serialization.md
+++ b/docs/dev/types_serialization.md
@@ -2,7 +2,8 @@
title: "Data Types & Serialization"
nav-id: types
nav-parent_id: dev
-nav-pos: 9
+nav-show_overview: true
+nav-pos: 50
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -61,7 +62,7 @@ The most frequent issues where users need to interact with Flink's data type han
by itself. Not all types are seamlessly handled by Kryo (and thus by Flink). For example, many Google Guava collection types do not work well
by default. The solution is to register additional serializers for the types that cause problems.
Call `.getConfig().addDefaultKryoSerializer(clazz, serializer)` on the `StreamExecutionEnvironment` or `ExecutionEnvironment`.
- Additional Kryo serializers are available in many libraries.
+ Additional Kryo serializers are available in many libraries. See [Custom Serializers]({{ site.baseurl }}/dev/custom_serializers) for more details on working with custom serializers.
* **Adding Type Hints:** Sometimes, when Flink cannot infer the generic types despits all tricks, a user must pass a *type hint*. That is generally
only necessary in the Java API. The [Type Hints Section](#type-hints-in-the-java-api) describes that in more detail.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/windows.md
----------------------------------------------------------------------
diff --git a/docs/dev/windows.md b/docs/dev/windows.md
index d6189d4..1170d0d 100644
--- a/docs/dev/windows.md
+++ b/docs/dev/windows.md
@@ -1,8 +1,8 @@
---
title: "Windows"
-nav-parent_id: dev
+nav-parent_id: streaming
nav-id: windows
-nav-pos: 3
+nav-pos: 10
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/examples/index.md
----------------------------------------------------------------------
diff --git a/docs/examples/index.md b/docs/examples/index.md
new file mode 100644
index 0000000..d04a1e9
--- /dev/null
+++ b/docs/examples/index.md
@@ -0,0 +1,39 @@
+---
+title: Examples
+nav-id: examples
+nav-title: '<i class="fa fa-file-code-o title appetizer" aria-hidden="true"></i> Examples'
+nav-parent_id: root
+nav-pos: 3
+nav-show_overview: true
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+[Sample Project in Java]({{ site.baseurl }}/quickstart/java_api_quickstart) and [Sample Project in Scala]({{ site.baseurl }}/quickstart/scala_api_quickstart) are guides to setting up Maven and SBT projects and include simple implementations of a word count application.
+
+[Monitoring Wikipedia Edits]({{ site.baseurl }}/quickstart/run_example_quickstart) is a more complete example of a streaming analytics application.
+
+[Building real-time dashboard applications with Apache Flink, Elasticsearch, and Kibana](https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana) is a blog post at elastic.co showing how to build a real-time dashboard solution for streaming data analytics using Apache Flink, Elasticsearch, and Kibana.
+
+The [Flink training website](http://dataartisans.github.io/flink-training) from data Artisans has a number of examples. See the hands-on sections, and the exercises.
+
+## Bundled Examples
+
+The Flink sources include a number of examples for both **streaming** ( [java](https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples) / [scala](https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples) ) and **batch** ( [java](https://github.com/apache/flink/tree/master/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java) / [scala](https://github.com/apache/flink/tree/master/flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala) ). These [instructions]({{ site.baseurl }}/dev/batch/examples.html#running-an-example) explain how to run the examples.
+
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index c40b17a..75b5328 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,7 +1,7 @@
---
title: "Apache Flink Documentation"
nav-pos: 0
-nav-title: '<i class="fa fa-home" aria-hidden="true"></i> Home'
+nav-title: '<i class="fa fa-home title" aria-hidden="true"></i> Home'
nav-parent_id: root
---
<!--
@@ -29,32 +29,8 @@ Apache Flink is an open source platform for distributed stream and batch data pr
## First Steps
-- **Concepts**: Start with the [basic concepts]({{ site.baseurl }}/concepts/index.html) of Flink. This will help you to fully understand the other parts of the documentation, including the setup and programming guides. It is highly recommended to read this first.
+- **Concepts**: Start with the basic concepts of Flink's [Dataflow Programming Model]({{ site.baseurl }}/concepts/programming-model.html) and [Distributed Runtime Environment]({{ site.baseurl }}/concepts/runtime.html). This will help you to fully understand the other parts of the documentation, including the setup and programming guides. It is highly recommended to read these sections first.
- **Quickstarts**: [Run an example program](quickstart/setup_quickstart.html) on your local machine or [write a simple program](quickstart/run_example_quickstart.html) working on live Wikipedia edits.
-- **Setup:** The [local]({{ site.baseurl }}/setup/local_setup.html), [cluster](setup/cluster_setup.html), and [cloud](setup/gce_setup.html) setup guides show you how to deploy Flink.
-
- **Programming Guides**: You can check out our guides about [basic concepts](dev/api_concepts.html) and the [DataStream API](dev/datastream_api.html) or [DataSet API](dev/batch/index.html) to learn how to write your first Flink programs.
-
-## Stack
-
-This is an overview of Flink's stack. Click on any component to go to the respective documentation page.
-
-<center>
- <img src="{{ site.baseurl }}/fig/stack.png" width="700px" alt="Apache Flink: Stack" usemap="#overview-stack">
-</center>
-
-<map name="overview-stack">
-<area id="lib-datastream-cep" title="CEP: Complex Event Processing" href="{{ site.baseurl }}/dev/libs/cep.html" shape="rect" coords="63,0,143,177" />
-<area id="lib-datastream-table" title="Table: Relational DataStreams" href="{{ site.baseurl }}/dev/table_api.html" shape="rect" coords="143,0,223,177" />
-<area id="lib-dataset-ml" title="FlinkML: Machine Learning" href="{{ site.baseurl }}/dev/libs/ml/index.html" shape="rect" coords="382,2,462,176" />
-<area id="lib-dataset-gelly" title="Gelly: Graph Processing" href="{{ site.baseurl }}/dev/libs/gelly/index.html" shape="rect" coords="461,0,541,177" />
-<area id="lib-dataset-table" title="Table API and SQL" href="{{ site.baseurl }}/dev/table_api.html" shape="rect" coords="544,0,624,177" />
-<area id="datastream" title="DataStream API" href="{{ site.baseurl }}/dev/datastream_api.html" shape="rect" coords="64,177,379,255" />
-<area id="dataset" title="DataSet API" href="{{ site.baseurl }}/dev/batch/index.html" shape="rect" coords="382,177,697,255" />
-<area id="runtime" title="Runtime" href="{{ site.baseurl }}/internals/general_arch.html" shape="rect" coords="63,257,700,335" />
-<area id="local" title="Local" href="{{ site.baseurl }}/setup/local_setup.html" shape="rect" coords="62,337,275,414" />
-<area id="cluster" title="Cluster" href="{{ site.baseurl }}/setup/cluster_setup.html" shape="rect" coords="273,336,486,413" />
-<area id="cloud" title="Cloud" href="{{ site.baseurl }}/setup/gce_setup.html" shape="rect" coords="485,336,700,414" />
-</map>
[4/4] flink git commit: [FLINK-5258] [docs] Reorganize the docs to
improve navigation and reduce duplication
Posted by se...@apache.org.
[FLINK-5258] [docs] Reorganize the docs to improve navigation and reduce duplication
This closes #2940
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/79d7e301
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/79d7e301
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/79d7e301
Branch: refs/heads/master
Commit: 79d7e3017efe7c96e449e6f339fd7184ef3d1ba2
Parents: e4c767a
Author: David Anderson <da...@alpinegizmo.com>
Authored: Fri Nov 25 15:44:21 2016 +0100
Committer: Stephan Ewen <se...@apache.org>
Committed: Wed Dec 14 14:56:33 2016 +0100
----------------------------------------------------------------------
docs/Gemfile | 10 +-
docs/Gemfile.lock | 72 ++--
docs/_includes/sidenav.html | 4 +-
docs/build_docs.sh | 6 +-
docs/concepts/index.md | 230 +-----------
docs/concepts/programming-model.md | 173 +++++++++
docs/concepts/runtime.md | 127 +++++++
docs/dev/api_concepts.md | 471 +------------------------
docs/dev/apis.md | 24 --
docs/dev/batch/examples.md | 12 +-
docs/dev/batch/index.md | 28 +-
docs/dev/batch/iterations.md | 8 +-
docs/dev/cluster_execution.md | 74 +---
docs/dev/connectors/cassandra.md | 2 +-
docs/dev/connectors/elasticsearch.md | 2 +-
docs/dev/connectors/elasticsearch2.md | 4 +-
docs/dev/connectors/filesystem_sink.md | 2 +-
docs/dev/connectors/guarantees.md | 143 ++++++++
docs/dev/connectors/index.md | 4 +-
docs/dev/connectors/kafka.md | 2 +-
docs/dev/connectors/kinesis.md | 2 +-
docs/dev/connectors/nifi.md | 2 +-
docs/dev/connectors/rabbitmq.md | 2 +-
docs/dev/connectors/redis.md | 2 +-
docs/dev/connectors/twitter.md | 2 +-
docs/dev/custom_serializers.md | 112 ++++++
docs/dev/datastream_api.md | 16 +-
docs/dev/event_time.md | 4 +-
docs/dev/execution.md | 24 ++
docs/dev/execution_configuration.md | 86 +++++
docs/dev/execution_plans.md | 80 +++++
docs/dev/index.md | 4 +-
docs/dev/java8.md | 4 +-
docs/dev/libraries.md | 2 +-
docs/dev/libs/cep.md | 4 +-
docs/dev/libs/gelly/index.md | 2 +-
docs/dev/libs/ml/index.md | 4 +-
docs/dev/libs/ml/quickstart.md | 2 +-
docs/dev/linking.md | 94 +++++
docs/dev/linking_with_flink.md | 146 ++++++++
docs/dev/local_execution.md | 4 +-
docs/dev/packaging.md | 77 ++++
docs/dev/parallel.md | 175 +++++++++
docs/dev/quickstarts.md | 24 --
docs/dev/scala_api_extensions.md | 4 +-
docs/dev/scala_shell.md | 6 +-
docs/dev/state.md | 76 +++-
docs/dev/state_backends.md | 4 +-
docs/dev/table_api.md | 6 +-
docs/dev/types_serialization.md | 5 +-
docs/dev/windows.md | 4 +-
docs/examples/index.md | 39 ++
docs/index.md | 28 +-
docs/internals/add_operator.md | 253 -------------
docs/internals/components.md | 59 ++++
docs/internals/general_arch.md | 101 ------
docs/internals/ide_setup.md | 55 +--
docs/internals/index.md | 5 +-
docs/internals/state_backends.md | 27 +-
docs/monitoring/best_practices.md | 94 +----
docs/monitoring/index.md | 6 +-
docs/page/css/flink.css | 21 ++
docs/quickstart/java_api_quickstart.md | 11 +-
docs/quickstart/run_example_quickstart.md | 8 +-
docs/quickstart/scala_api_quickstart.md | 8 +-
docs/quickstart/setup_quickstart.md | 225 ++++++++----
docs/redirects/concepts.md | 2 +-
docs/setup/building.md | 6 +-
docs/setup/checkpoints.md | 57 +++
docs/setup/cluster_setup.md | 60 ++--
docs/setup/config.md | 11 +-
docs/setup/fault_tolerance.md | 250 ++-----------
docs/setup/flink_on_windows.md | 86 +++++
docs/setup/index.md | 6 +-
docs/setup/local_setup.md | 153 --------
docs/setup/savepoints.md | 13 +-
docs/setup/security-ssl.md | 2 +-
docs/start/index.md | 26 ++
78 files changed, 2030 insertions(+), 1959 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/Gemfile
----------------------------------------------------------------------
diff --git a/docs/Gemfile b/docs/Gemfile
index b7a974c..c1a1555 100644
--- a/docs/Gemfile
+++ b/docs/Gemfile
@@ -17,11 +17,13 @@
################################################################################
source 'https://rubygems.org'
-
-ruby '>=1.9.0'
+ruby '~> 2.3.0'
# Dependencies required to build the Flink docs
-gem 'jekyll', '2.5.3'
-gem 'kramdown', '1.10.0'
+gem 'jekyll', '~> 3.3.0'
+gem 'kramdown', '~> 1.13.0'
gem 'pygments.rb', '0.6.3'
gem 'therubyracer', '0.12.2'
+group :jekyll_plugins do
+ gem 'hawkins'
+end
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/Gemfile.lock
----------------------------------------------------------------------
diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock
index bf86631..da7e04a 100644
--- a/docs/Gemfile.lock
+++ b/docs/Gemfile.lock
@@ -1,83 +1,67 @@
GEM
remote: https://rubygems.org/
specs:
- addressable (2.4.0)
- blankslate (2.1.2.4)
- classifier-reborn (2.0.4)
- fast-stemmer (~> 1.0)
- coffee-script (2.4.1)
- coffee-script-source
- execjs
- coffee-script-source (1.11.1)
- colorator (0.1)
- execjs (2.7.0)
- faraday (0.9.2)
- multipart-post (>= 1.2, < 3)
- fast-stemmer (1.0.2)
+ addressable (2.5.0)
+ public_suffix (~> 2.0, >= 2.0.2)
+ colorator (1.1.0)
+ em-websocket (0.5.1)
+ eventmachine (>= 0.12.9)
+ http_parser.rb (~> 0.6.0)
+ eventmachine (1.2.1)
ffi (1.9.14)
- jekyll (2.5.3)
- classifier-reborn (~> 2.0)
- colorator (~> 0.1)
- jekyll-coffeescript (~> 1.0)
- jekyll-gist (~> 1.0)
- jekyll-paginate (~> 1.0)
+ forwardable-extended (2.6.0)
+ hawkins (2.0.4)
+ em-websocket (~> 0.5)
+ jekyll (~> 3.1)
+ http_parser.rb (0.6.0)
+ jekyll (3.3.1)
+ addressable (~> 2.4)
+ colorator (~> 1.0)
jekyll-sass-converter (~> 1.0)
jekyll-watch (~> 1.1)
kramdown (~> 1.3)
- liquid (~> 2.6.1)
+ liquid (~> 3.0)
mercenary (~> 0.3.3)
- pygments.rb (~> 0.6.0)
- redcarpet (~> 3.1)
+ pathutil (~> 0.9)
+ rouge (~> 1.7)
safe_yaml (~> 1.0)
- toml (~> 0.1.0)
- jekyll-coffeescript (1.0.1)
- coffee-script (~> 2.2)
- jekyll-gist (1.4.0)
- octokit (~> 4.3.0)
- jekyll-paginate (1.1.0)
jekyll-sass-converter (1.5.0)
sass (~> 3.4)
jekyll-watch (1.5.0)
listen (~> 3.0, < 3.1)
- kramdown (1.10.0)
+ kramdown (1.13.1)
libv8 (3.16.14.17)
- liquid (2.6.3)
+ liquid (3.0.6)
listen (3.0.8)
rb-fsevent (~> 0.9, >= 0.9.4)
rb-inotify (~> 0.9, >= 0.9.7)
mercenary (0.3.6)
- multipart-post (2.0.0)
- octokit (4.3.0)
- sawyer (~> 0.7.0, >= 0.5.3)
- parslet (1.5.0)
- blankslate (~> 2.0)
+ pathutil (0.14.0)
+ forwardable-extended (~> 2.6)
posix-spawn (0.3.12)
+ public_suffix (2.0.4)
pygments.rb (0.6.3)
posix-spawn (~> 0.3.6)
yajl-ruby (~> 1.2.0)
rb-fsevent (0.9.8)
rb-inotify (0.9.7)
ffi (>= 0.5.0)
- redcarpet (3.3.4)
ref (2.0.0)
+ rouge (1.11.1)
safe_yaml (1.0.4)
sass (3.4.22)
- sawyer (0.7.0)
- addressable (>= 2.3.5, < 2.5)
- faraday (~> 0.8, < 0.10)
therubyracer (0.12.2)
libv8 (~> 3.16.14.0)
ref
- toml (0.1.2)
- parslet (~> 1.5.0)
yajl-ruby (1.2.1)
PLATFORMS
ruby
DEPENDENCIES
- jekyll (= 2.5.3)
- kramdown (= 1.10.0)
+ hawkins
+ jekyll (~> 3.3.0)
+ kramdown (~> 1.13.0)
pygments.rb (= 0.6.3)
therubyracer (= 0.12.2)
@@ -85,4 +69,4 @@ RUBY VERSION
ruby 2.3.1p112
BUNDLED WITH
- 1.13.2
+ 1.13.6
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/_includes/sidenav.html
----------------------------------------------------------------------
diff --git a/docs/_includes/sidenav.html b/docs/_includes/sidenav.html
index b56bcf2..e3bb2d7 100644
--- a/docs/_includes/sidenav.html
+++ b/docs/_includes/sidenav.html
@@ -101,6 +101,8 @@ level is determined by 'nav-pos'.
{% capture target %}"{{ site.baseurl }}{{ this.url }}"{% if active %} class="active"{% endif %}{% endcapture %}
{% capture overview_target %}"{{ site.baseurl }}{{ this.url }}"{% if this.url == page.url %} class="active"{% endif %}{% endcapture %}
+ {% if this.section-break %}<hr class="section-break"></hr>{% endif %}
+
{% assign pos = pos | plus: 1 %}
{% if this.nav-id %}
{% assign children = (site.pages | where: "nav-parent_id" , this.nav-id | sort: "nav-pos") %}
@@ -124,7 +126,7 @@ level is determined by 'nav-pos'.
{% endif %}
{% endfor %}
<li class="divider"></li>
- <li><a href="http://flink.apache.org"><i class="fa fa-external-link" aria-hidden="true"></i> Project Page</a></li>
+ <li><a href="http://flink.apache.org"><i class="fa fa-external-link title" aria-hidden="true"></i> Project Page</a></li>
</ul>
<div class="sidenav-search-box">
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/build_docs.sh
----------------------------------------------------------------------
diff --git a/docs/build_docs.sh b/docs/build_docs.sh
index 2981c4a..df83ac4 100755
--- a/docs/build_docs.sh
+++ b/docs/build_docs.sh
@@ -46,11 +46,15 @@ DOCS_DST=${DOCS_SRC}/content
JEKYLL_CMD="build"
# if -p flag is provided, serve site on localhost
-while getopts ":p" opt; do
+# -i is like -p, but incremental (which has some issues, but is very fast)
+while getopts ":p:i" opt; do
case $opt in
p)
JEKYLL_CMD="serve --baseurl= --watch"
;;
+ i)
+ JEKYLL_CMD="liveserve --baseurl= --watch --incremental"
+ ;;
esac
done
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/concepts/index.md
----------------------------------------------------------------------
diff --git a/docs/concepts/index.md b/docs/concepts/index.md
index a9638451..25565e8 100644
--- a/docs/concepts/index.md
+++ b/docs/concepts/index.md
@@ -2,8 +2,12 @@
title: Concepts
nav-id: concepts
nav-pos: 1
-nav-title: '<i class="fa fa-map-o" aria-hidden="true"></i> Concepts'
+nav-title: '<i class="fa fa-map-o title appetizer" aria-hidden="true"></i> Concepts'
nav-parent_id: root
+section-break: true
+layout: redirect
+redirect: /concepts/programming-model.html
+permalink: /concepts/index.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -23,227 +27,3 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
-
-* This will be replaced by the TOC
-{:toc}
-
-## Programs and Dataflows
-
-The basic building blocks of Flink programs are **streams** and **transformations** (note that a DataSet is internally
-also a stream). A *stream* is an intermediate result, and a *transformation* is an operation that takes one or more streams
-as input, and computes one or more result streams from them.
-
-When executed, Flink programs are mapped to **streaming dataflows**, consisting of **streams** and transformation **operators**.
-Each dataflow starts with one or more **sources** and ends in one or more **sinks**. The dataflows may resemble
-arbitrary **directed acyclic graphs** *(DAGs)*. (Special forms of cycles are permitted via *iteration* constructs, we
-omit this here for simplicity).
-
-In most cases, there is a one-to-one correspondence between the transformations in the programs and the operators
-in the dataflow. Sometimes, however, one transformation may consist of multiple transformation operators.
-
-<img src="{{ site.baseurl }}/fig/program_dataflow.svg" alt="A DataStream program, and its dataflow." class="offset" width="80%" />
-
-{% top %}
-
-### Parallel Dataflows
-
-Programs in Flink are inherently parallel and distributed. *Streams* are split into **stream partitions** and
-*operators* are split into **operator subtasks**. The operator subtasks execute independently from each other,
-in different threads and on different machines or containers.
-
-The number of operator subtasks is the **parallelism** of that particular operator. The parallelism of a stream
-is always that of its producing operator. Different operators of the program may have a different parallelism.
-
-<img src="{{ site.baseurl }}/fig/parallel_dataflow.svg" alt="A parallel dataflow" class="offset" width="80%" />
-
-Streams can transport data between two operators in a *one-to-one* (or *forwarding*) pattern, or in a *redistributing* pattern:
-
- - **One-to-one** streams (for example between the *source* and the *map()* operators) preserves partitioning and order of
- elements. That means that subtask[1] of the *map()* operator will see the same elements in the same order, as they
- were produced by subtask[1] of the *source* operator.
-
- - **Redistributing** streams (between *map()* and *keyBy/window*, as well as between *keyBy/window* and *sink*) change
- the partitioning of streams. Each *operator subtask* sends data to different target subtasks,
- depending on the selected transformation. Examples are *keyBy()* (re-partitions by hash code), *broadcast()*, or
- *rebalance()* (random redistribution).
- In a *redistributing* exchange, order among elements is only preserved for each pair of sending- and receiving
- task (for example subtask[1] of *map()* and subtask[2] of *keyBy/window*).
-
-{% top %}
-
-### Tasks & Operator Chains
-
-For distributed execution, Flink *chains* operator subtasks together into *tasks*. Each task is executed by one thread.
-Chaining operators together into tasks is a useful optimization: it reduces the overhead of thread-to-thread
-handover and buffering, and increases overall throughput while decreasing latency.
-The chaining behavior can be configured in the APIs.
-
-The sample dataflow in the figure below is executed with five subtasks, and hence with five parallel threads.
-
-<img src="{{ site.baseurl }}/fig/tasks_chains.svg" alt="Operator chaining into Tasks" class="offset" width="80%" />
-
-{% top %}
-
-## Distributed Execution
-
-**Master, Worker, Client**
-
-The Flink runtime consists of two types of processes:
-
- - The **master** processes (also called *JobManagers*) coordinate the distributed execution. They schedule tasks, coordinate
- checkpoints, coordinate recovery on failures, etc.
-
- There is always at least one master process. A high-availability setup will have multiple master processes, out of
- which one is always the *leader*, and the others are *standby*.
-
- - The **worker** processes (also called *TaskManagers*) execute the *tasks* (or more specifically, the subtasks) of a dataflow,
- and buffer and exchange the data *streams*.
-
- There must always be at least one worker process.
-
-The master and worker processes can be started in an arbitrary fashion: Directly on the machines, via containers, or via
-resource frameworks like YARN. Workers connect to masters, announcing themselves as available, and get work assigned.
-
-The **client** is not part of the runtime and program execution, but is used to prepare and send a dataflow to the master.
-After that, the client can disconnect, or stay connected to receive progress reports. The client runs either as part of the
-Java/Scala program that triggers the execution, or in the command line process `./bin/flink run ...`.
-
-<img src="{{ site.baseurl }}/fig/processes.svg" alt="The processes involved in executing a Flink dataflow" class="offset" width="80%" />
-
-{% top %}
-
-### Workers, Slots, Resources
-
-Each worker (TaskManager) is a *JVM process*, and may execute one or more subtasks in separate threads.
-To control how many tasks a worker accepts, a worker has so called **task slots** (at least one).
-
-Each *task slot* represents a fixed subset of resources of the TaskManager. A TaskManager with three slots, for example,
-will dedicate 1/3 of its managed memory to each slot. Slotting the resources means that a subtask will not
-compete with subtasks from other jobs for managed memory, but instead has a certain amount of reserved
-managed memory. Note that no CPU isolation happens here, slots currently only separate managed memory of tasks.
-
-Adjusting the number of task slots thus allows users to define how subtasks are isolated against each other.
-Having one slot per TaskManager means each task group runs in a separate JVM (which can be started in a
-separate container, for example). Having multiple slots
-means more subtasks share the same JVM. Tasks in the same JVM share TCP connections (via multiplexing) and
-heartbeats messages. They may also share data sets and data structures, thus reducing the per-task overhead.
-
-<img src="{{ site.baseurl }}/fig/tasks_slots.svg" alt="A TaskManager with Task Slots and Tasks" class="offset" width="80%" />
-
-By default, Flink allows subtasks to share slots, if they are subtasks of different tasks, but from the same
-job. The result is that one slot may hold an entire pipeline of the job. Allowing this *slot sharing*
-has two main benefits:
-
- - A Flink cluster needs exactly as many tasks slots, as the highest parallelism used in the job.
- No need to calculate how many tasks (with varying parallelism) a program contains in total.
-
- - It is easier to get better resource utilization. Without slot sharing, the non-intensive
- *source/map()* subtasks would block as many resources as the resource intensive *window* subtasks.
- With slot sharing, increasing the base parallelism from two to six yields full utilization of the
- slotted resources, while still making sure that each TaskManager gets only a fair share of the
- heavy subtasks.
-
-The slot sharing behavior can be controlled in the APIs, to prevent sharing where it is undesirable.
-The mechanism for that are the *resource groups*, which define what (sub)tasks may share slots.
-
-As a rule-of-thumb, a good default number of task slots would be the number of CPU cores.
-With hyper threading, each slot then takes 2 or more hardware thread contexts.
-
-<img src="{{ site.baseurl }}/fig/slot_sharing.svg" alt="TaskManagers with shared Task Slots" class="offset" width="80%" />
-
-{% top %}
-
-## Time and Windows
-
-Aggregating events (e.g., counts, sums) works slightly differently on streams than in batch processing.
-For example, it is impossible to first count all elements in the stream and then return the count,
-because streams are in general infinite (unbounded). Instead, aggregates on streams (counts, sums, etc),
-are scoped by **windows**, such as *"count over the last 5 minutes"*, or *"sum of the last 100 elements"*.
-
-Windows can be *time driven* (example: every 30 seconds) or *data driven* (example: every 100 elements).
-One typically distinguishes different types of windows, such as *tumbling windows* (no overlap),
-*sliding windows* (with overlap), and *session windows* (gap of activity).
-
-<img src="{{ site.baseurl }}/fig/windows.svg" alt="Time- and Count Windows" class="offset" width="80%" />
-
-More window examples can be found in this [blog post](https://flink.apache.org/news/2015/12/04/Introducing-windows.html).
-
-{% top %}
-
-### Time
-
-When referring to time in a streaming program (for example to define windows), one can refer to different notions
-of time:
-
- - **Event Time** is the time when an event was created. It is usually described by a timestamp in the events,
- for example attached by the producing sensor, or the producing service. Flink accesses event timestamps
- via [timestamp assigners]({{ site.baseurl }}/dev/event_timestamps_watermarks.html).
-
- - **Ingestion time** is the time when an event enters the Flink dataflow at the source operator.
-
- - **Processing Time** is the local time at each operator that performs a time-based operation.
-
-<img src="{{ site.baseurl }}/fig/event_ingestion_processing_time.svg" alt="Event Time, Ingestion Time, and Processing Time" class="offset" width="80%" />
-
-More details on how to handle time are in the [event time docs]({{ site.baseurl }}/dev/event_time.html).
-
-{% top %}
-
-## State and Fault Tolerance
-
-While many operations in a dataflow simply look at one individual *event at a time* (for example an event parser),
-some operations remember information across individual events (for example window operators).
-These operations are called **stateful**.
-
-The state of stateful operations is maintained in what can be thought of as an embedded key/value store.
-The state is partitioned and distributed strictly together with the streams that are read by the
-stateful operators. Hence, access the key/value state is only possible on *keyed streams*, after a *keyBy()* function,
-and is restricted to the values of the current event's key. Aligning the keys of streams and state
-makes sure that all state updates are local operations, guaranteeing consistency without transaction overhead.
-This alignment also allows Flink to redistribute the state and adjust the stream partitioning transparently.
-
-<img src="{{ site.baseurl }}/fig/state_partitioning.svg" alt="State and Partitioning" class="offset" width="50%" />
-
-{% top %}
-
-### Checkpoints for Fault Tolerance
-
-Flink implements fault tolerance using a combination of **stream replay** and **checkpoints**. A checkpoint
-defines a consistent point in streams and state from which a streaming dataflow can resume, and maintain consistency
-*(exactly-once processing semantics)*. The events and state updates since the last checkpoint are replayed from the input streams.
-
-The checkpoint interval is a means of trading off the overhead of fault tolerance during execution, with the recovery time (the amount
-of events that need to be replayed).
-
-More details on checkpoints and fault tolerance are in the [fault tolerance docs]({{ site.baseurl }}/internals/stream_checkpointing.html).
-
-<img src="{{ site.baseurl }}/fig/checkpoints.svg" alt="checkpoints and snapshots" class="offset" width="60%" />
-
-{% top %}
-
-### State Backends
-
-The exact data structures in which the key/values indexes are stored depend on the chosen **state backend**. One state backend
-stores data in an in-memory hash map, another state backend uses [RocksDB](http://rocksdb.org) as the key/value index.
-In addition to defining the data structure that holds the state, the state backends also implements the logic to
-take a point-in-time snapshot of the key/value state and store that snapshot as part of a checkpoint.
-
-{% top %}
-
-## Batch on Streaming
-
-Flink executes batch programs as a special case of streaming programs, where the streams are bounded (finite number of elements).
-A *DataSet* is treated internally as a stream of data. The concepts above thus apply to batch programs in the
-same way as well as they apply to streaming programs, with minor exceptions:
-
- - Programs in the DataSet API do not use checkpoints. Recovery happens by fully replaying the streams.
- That is possible, because inputs are bounded. This pushes the cost more towards the recovery,
- but makes the regular processing cheaper, because it avoids checkpoints.
-
- - Stateful operation in the DataSet API use simplified in-memory/out-of-core data structures, rather than
- key/value indexes.
-
- - The DataSet API introduces special synchronized (superstep-based) iterations, which are only possible on
- bounded streams. For details, check out the [iteration docs]({{ site.baseurl }}/dev/batch/iterations.html).
-
-{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/concepts/programming-model.md
----------------------------------------------------------------------
diff --git a/docs/concepts/programming-model.md b/docs/concepts/programming-model.md
new file mode 100644
index 0000000..5ab6b8f
--- /dev/null
+++ b/docs/concepts/programming-model.md
@@ -0,0 +1,173 @@
+---
+title: Dataflow Programming Model
+nav-id: programming-model
+nav-pos: 1
+nav-title: Programming Model
+nav-parent_id: concepts
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* This will be replaced by the TOC
+{:toc}
+
+## Programs and Dataflows
+
+The basic building blocks of Flink programs are **streams** and **transformations**. (Note that the
+DataSets used in Flink's batch API are also streams internally -- more about that
+later.) Conceptually a *stream* is a never-ending flow of data records, and a *transformation* is an
+operation that takes one or more streams as input, and produces one or more output streams as a
+result.
+
+When executed, Flink programs are mapped to **streaming dataflows**, consisting of **streams** and transformation **operators**.
+Each dataflow starts with one or more **sources** and ends in one or more **sinks**. The dataflows resemble
+arbitrary **directed acyclic graphs** *(DAGs)*. Although special forms of cycles are permitted via
+*iteration* constructs, for the most part we will gloss over this for simplicity.
+
+<img src="{{ site.baseurl }}/fig/program_dataflow.svg" alt="A DataStream program, and its dataflow." class="offset" width="80%" />
+
+Often there is a one-to-one correspondence between the transformations in the programs and the operators
+in the dataflow. Sometimes, however, one transformation may consist of multiple transformation operators.
+
+{% top %}
+
+## Parallel Dataflows
+
+Programs in Flink are inherently parallel and distributed. This parallelism is expressed in Flink's
+DataStream API with the *keyBy()* operator, which can be thought of as a declaration that the stream can
+be operated on in parallel for different values of the key.
+
+*Streams* are split into **stream partitions**, and *operators* are split into **operator
+subtasks**. The operator subtasks are independent of one another, and execute in different threads
+and possibly on different machines or containers.
+
+The number of operator subtasks is the **parallelism** of that particular operator. The parallelism of a stream
+is always that of its producing operator. Different operators of the same program may have different
+levels of parallelism.
+
+<img src="{{ site.baseurl }}/fig/parallel_dataflow.svg" alt="A parallel dataflow" class="offset" width="80%" />
+
+Streams can transport data between two operators in a *one-to-one* (or *forwarding*) pattern, or in a *redistributing* pattern:
+
+ - **One-to-one** streams (for example between the *Source* and the *map()* operators in the figure
+ above) preserve the partitioning and ordering of the
+ elements. That means that subtask[1] of the *map()* operator will see the same elements in the same order as they
+ were produced by subtask[1] of the *Source* operator.
+
+ - **Redistributing** streams (as between *map()* and *keyBy/window* above, as well as between
+ *keyBy/window* and *Sink*) change the partitioning of streams. Each *operator subtask* sends
+ data to different target subtasks, depending on the selected transformation. Examples are
+ *keyBy()* (which re-partitions by hashing the key), *broadcast()*, or *rebalance()* (which
+ re-partitions randomly). In a *redistributing* exchange the ordering among the elements is
+ only preserved within each pair of sending and receiving subtasks (for example, subtask[1]
+ of *map()* and subtask[2] of *keyBy/window*). So in this example, the ordering within each key
+ is preserved, but the parallelism does introduce non-determinism regarding the order in
+ which the aggregated results for different keys arrive at the sink.
+
+{% top %}
+
+## Windows
+
+Aggregating events (e.g., counts, sums) works differently on streams than in batch processing.
+For example, it is impossible to count all elements in a stream,
+because streams are in general infinite (unbounded). Instead, aggregates on streams (counts, sums, etc),
+are scoped by **windows**, such as *"count over the last 5 minutes"*, or *"sum of the last 100 elements"*.
+
+Windows can be *time driven* (example: every 30 seconds) or *data driven* (example: every 100 elements).
+One typically distinguishes different types of windows, such as *tumbling windows* (no overlap),
+*sliding windows* (with overlap), and *session windows* (punctuated by a gap of inactivity).
+
+<img src="{{ site.baseurl }}/fig/windows.svg" alt="Time- and Count Windows" class="offset" width="80%" />
+
+More window examples can be found in this [blog post](https://flink.apache.org/news/2015/12/04/Introducing-windows.html).
+
+{% top %}
+
+## Time
+
+When referring to time in a streaming program (for example to define windows), one can refer to different notions
+of time:
+
+ - **Event Time** is the time when an event was created. It is usually described by a timestamp in the events,
+ for example attached by the producing sensor, or the producing service. Flink accesses event timestamps
+ via [timestamp assigners]({{ site.baseurl }}/dev/event_timestamps_watermarks.html).
+
+ - **Ingestion time** is the time when an event enters the Flink dataflow at the source operator.
+
+ - **Processing Time** is the local time at each operator that performs a time-based operation.
+
+<img src="{{ site.baseurl }}/fig/event_ingestion_processing_time.svg" alt="Event Time, Ingestion Time, and Processing Time" class="offset" width="80%" />
+
+More details on how to handle time are in the [event time docs]({{ site.baseurl }}/dev/event_time.html).
+
+{% top %}
+
+## Stateful Operations
+
+While many operations in a dataflow simply look at one individual *event at a time* (for example an event parser),
+some operations remember information across multiple events (for example window operators).
+These operations are called **stateful**.
+
+The state of stateful operations is maintained in what can be thought of as an embedded key/value store.
+The state is partitioned and distributed strictly together with the streams that are read by the
+stateful operators. Hence, access to the key/value state is only possible on *keyed streams*, after a *keyBy()* function,
+and is restricted to the values associated with the current event's key. Aligning the keys of streams and state
+makes sure that all state updates are local operations, guaranteeing consistency without transaction overhead.
+This alignment also allows Flink to redistribute the state and adjust the stream partitioning transparently.
+
+<img src="{{ site.baseurl }}/fig/state_partitioning.svg" alt="State and Partitioning" class="offset" width="50%" />
+
+{% top %}
+
+## Checkpoints for Fault Tolerance
+
+Flink implements fault tolerance using a combination of **stream replay** and **checkpointing**. A
+checkpoint is related to a specific point in each of the input streams along with the corresponding state for each
+of the operators. A streaming dataflow can be resumed from a checkpoint while maintaining consistency *(exactly-once
+processing semantics)* by restoring the state of the operators and replaying the events from the
+point of the checkpoint.
+
+The checkpoint interval is a means of trading off the overhead of fault tolerance during execution with the recovery time (the number
+of events that need to be replayed).
+
+More details on checkpoints and fault tolerance are in the [fault tolerance docs]({{ site.baseurl }}/internals/stream_checkpointing.html).
+
+{% top %}
+
+## Batch on Streaming
+
+Flink executes batch programs as a special case of streaming programs, where the streams are bounded (finite number of elements).
+A *DataSet* is treated internally as a stream of data. The concepts above thus apply to batch programs in the
+same way as well as they apply to streaming programs, with minor exceptions:
+
+ - Programs in the DataSet API do not use checkpoints. Recovery happens by fully replaying the streams.
+ That is possible, because inputs are bounded. This pushes the cost more towards the recovery,
+ but makes the regular processing cheaper, because it avoids checkpoints.
+
+ - Stateful operations in the DataSet API use simplified in-memory/out-of-core data structures, rather than
+ key/value indexes.
+
+ - The DataSet API introduces special synchronized (superstep-based) iterations, which are only possible on
+ bounded streams. For details, check out the [iteration docs]({{ site.baseurl }}/dev/batch/iterations.html).
+
+{% top %}
+
+## Next Steps
+
+Continue with the basic concepts in Flink's [Distributed Runtime]({{ site.baseurl }}/concepts/runtime).
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/concepts/runtime.md
----------------------------------------------------------------------
diff --git a/docs/concepts/runtime.md b/docs/concepts/runtime.md
new file mode 100644
index 0000000..016861a
--- /dev/null
+++ b/docs/concepts/runtime.md
@@ -0,0 +1,127 @@
+---
+title: Distributed Runtime Environment
+nav-pos: 2
+nav-title: Distributed Runtime
+nav-parent_id: concepts
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* This will be replaced by the TOC
+{:toc}
+
+## Tasks and Operator Chains
+
+For distributed execution, Flink *chains* operator subtasks together into *tasks*. Each task is executed by one thread.
+Chaining operators together into tasks is a useful optimization: it reduces the overhead of thread-to-thread
+handover and buffering, and increases overall throughput while decreasing latency.
+The chaining behavior can be configured in the APIs.
+
+The sample dataflow in the figure below is executed with five subtasks, and hence with five parallel threads.
+
+<img src="{{ site.baseurl }}/fig/tasks_chains.svg" alt="Operator chaining into Tasks" class="offset" width="80%" />
+
+{% top %}
+
+## Job Managers, Task Managers, Clients
+
+The Flink runtime consists of two types of processes:
+
+ - The **JobManagers** (also called *masters*) coordinate the distributed execution. They schedule tasks, coordinate
+ checkpoints, coordinate recovery on failures, etc.
+
+ There is always at least one Job Manager. A high-availability setup will have multiple JobManagers, one of
+ which one is always the *leader*, and the others are *standby*.
+
+ - The **TaskManagers** (also called *workers*) execute the *tasks* (or more specifically, the subtasks) of a dataflow,
+ and buffer and exchange the data *streams*.
+
+ There must always be at least one TaskManager.
+
+The JobManagers and TaskManagers can be started in various ways: directly on the machines, in
+containers, or managed by resource frameworks like YARN. TaskManagers connect to JobManagers, announcing
+themselves as available, and are assigned work.
+
+The **client** is not part of the runtime and program execution, but is used to prepare and send a dataflow to the JobManager.
+After that, the client can disconnect, or stay connected to receive progress reports. The client runs either as part of the
+Java/Scala program that triggers the execution, or in the command line process `./bin/flink run ...`.
+
+<img src="{{ site.baseurl }}/fig/processes.svg" alt="The processes involved in executing a Flink dataflow" class="offset" width="80%" />
+
+{% top %}
+
+## Task Slots and Resources
+
+Each worker (TaskManager) is a *JVM process*, and may execute one or more subtasks in separate threads.
+To control how many tasks a worker accepts, a worker has so called **task slots** (at least one).
+
+Each *task slot* represents a fixed subset of resources of the TaskManager. A TaskManager with three slots, for example,
+will dedicate 1/3 of its managed memory to each slot. Slotting the resources means that a subtask will not
+compete with subtasks from other jobs for managed memory, but instead has a certain amount of reserved
+managed memory. Note that no CPU isolation happens here; currently slots only separate the managed memory of tasks.
+
+By adjusting the number of task slots, users can define how subtasks are isolated from each other.
+Having one slot per TaskManager means each task group runs in a separate JVM (which can be started in a
+separate container, for example). Having multiple slots
+means more subtasks share the same JVM. Tasks in the same JVM share TCP connections (via multiplexing) and
+heartbeat messages. They may also share data sets and data structures, thus reducing the per-task overhead.
+
+<img src="{{ site.baseurl }}/fig/tasks_slots.svg" alt="A TaskManager with Task Slots and Tasks" class="offset" width="80%" />
+
+By default, Flink allows subtasks to share slots even if they are subtasks of different tasks, so long as
+they are from the same job. The result is that one slot may hold an entire pipeline of the
+job. Allowing this *slot sharing* has two main benefits:
+
+ - A Flink cluster needs exactly as many task slots as the highest parallelism used in the job.
+ No need to calculate how many tasks (with varying parallelism) a program contains in total.
+
+ - It is easier to get better resource utilization. Without slot sharing, the non-intensive
+ *source/map()* subtasks would block as many resources as the resource intensive *window* subtasks.
+ With slot sharing, increasing the base parallelism in our example from two to six yields full utilization of the
+ slotted resources, while making sure that the heavy subtasks are fairly distributed among the TaskManagers.
+
+<img src="{{ site.baseurl }}/fig/slot_sharing.svg" alt="TaskManagers with shared Task Slots" class="offset" width="80%" />
+
+The APIs also include a *resource group* mechanism which can be used to prevent undesirable slot sharing.
+
+As a rule-of-thumb, a good default number of task slots would be the number of CPU cores.
+With hyper-threading, each slot then takes 2 or more hardware thread contexts.
+
+{% top %}
+
+## State Backends
+
+The exact data structures in which the key/values indexes are stored depends on the chosen **state backend**. One state backend
+stores data in an in-memory hash map, another state backend uses [RocksDB](http://rocksdb.org) as the key/value store.
+In addition to defining the data structure that holds the state, the state backends also implement the logic to
+take a point-in-time snapshot of the key/value state and store that snapshot as part of a checkpoint.
+
+<img src="{{ site.baseurl }}/fig/checkpoints.svg" alt="checkpoints and snapshots" class="offset" width="60%" />
+
+{% top %}
+
+## Savepoints
+
+Programs written in the Data Stream API can resume execution from a **savepoint**. Savepoints allow both updating your programs and your Flink cluster without losing any state.
+
+Savepoints are **manually triggered checkpoints**, which take a snapshot of the program and write it out to a state backend. They rely on the regular checkpointing mechanism for this. During execution programs are periodically snapshotted on the worker nodes and produce checkpoints. For recovery only the last completed checkpoint is needed and older checkpoints can be safely discarded as soon as a new one is completed.
+
+Savepoints are similar to these periodic checkpoints except that they are **triggered by the user** and **don't automatically expire** when newer checkpoints are completed.
+
+{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/api_concepts.md
----------------------------------------------------------------------
diff --git a/docs/dev/api_concepts.md b/docs/dev/api_concepts.md
index 35e4d3a..aa064d0 100644
--- a/docs/dev/api_concepts.md
+++ b/docs/dev/api_concepts.md
@@ -1,7 +1,9 @@
---
title: "Basic API Concepts"
-nav-parent_id: apis
+nav-parent_id: dev
nav-pos: 1
+nav-show_overview: true
+nav-id: api-concepts
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -24,15 +26,15 @@ under the License.
Flink programs are regular programs that implement transformations on distributed collections
(e.g., filtering, mapping, updating state, joining, grouping, defining windows, aggregating).
-Collections are initially created from sources (e.g., by reading files, kafka, or from local
+Collections are initially created from sources (e.g., by reading from files, kafka topics, or from local, in-memory
collections). Results are returned via sinks, which may for example write the data to
-(distributed) files, or to standard output (for example the command line terminal).
+(distributed) files, or to standard output (for example, the command line terminal).
Flink programs run in a variety of contexts, standalone, or embedded in other programs.
The execution can happen in a local JVM, or on clusters of many machines.
-Depending on the type of data sources, i.e. bounded or unbounded sources you would either
-write a batch program or a streaming program where the DataSet API is used for the former
-and the DataStream API is used for the latter. This guide will introduce the basic concepts
+Depending on the type of data sources, i.e. bounded or unbounded sources, you would either
+write a batch program or a streaming program where the DataSet API is used for batch
+and the DataStream API is used for streaming. This guide will introduce the basic concepts
that are common to both APIs but please see our
[Streaming Guide]({{ site.baseurl }}/dev/datastream_api.html) and
[Batch Guide]({{ site.baseurl }}/dev/batch/index.html) for concrete information about
@@ -45,129 +47,6 @@ in the `DataSet` API, just replace by `ExecutionEnvironment` and `DataSet`.
* This will be replaced by the TOC
{:toc}
-Linking with Flink
-------------------
-
-To write programs with Flink, you need to include the Flink library corresponding to
-your programming language in your project.
-
-The simplest way to do this is to use one of the quickstart scripts: either for
-[Java]({{ site.baseurl }}/quickstart/java_api_quickstart.html) or for [Scala]({{ site.baseurl }}/quickstart/scala_api_quickstart.html). They
-create a blank project from a template (a Maven Archetype), which sets up everything for you. To
-manually create the project, you can use the archetype and create a project by calling:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight bash %}
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.flink \
- -DarchetypeArtifactId=flink-quickstart-java \
- -DarchetypeVersion={{site.version }}
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight bash %}
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.flink \
- -DarchetypeArtifactId=flink-quickstart-scala \
- -DarchetypeVersion={{site.version }}
-{% endhighlight %}
-</div>
-</div>
-
-The archetypes are working for stable releases and preview versions (`-SNAPSHOT`).
-
-If you want to add Flink to an existing Maven project, add the following entry to your
-*dependencies* section in the *pom.xml* file of your project:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight xml %}
-<!-- Use this dependency if you are using the DataStream API -->
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-<!-- Use this dependency if you are using the DataSet API -->
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight xml %}
-<!-- Use this dependency if you are using the DataStream API -->
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-<!-- Use this dependency if you are using the DataSet API -->
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-{% endhighlight %}
-
-**Important:** When working with the Scala API you must have one of these two imports:
-{% highlight scala %}
-import org.apache.flink.api.scala._
-{% endhighlight %}
-
-or
-
-{% highlight scala %}
-import org.apache.flink.api.scala.createTypeInformation
-{% endhighlight %}
-
-The reason is that Flink analyzes the types that are used in a program and generates serializers
-and comparaters for them. By having either of those imports you enable an implicit conversion
-that creates the type information for Flink operations.
-</div>
-</div>
-
-#### Scala Dependency Versions
-
-Because Scala 2.10 binary is not compatible with Scala 2.11 binary, we provide multiple artifacts
-to support both Scala versions.
-
-Starting from the 0.10 line, we cross-build all Flink modules for both 2.10 and 2.11. If you want
-to run your program on Flink with Scala 2.11, you need to add a `_2.11` suffix to the `artifactId`
-values of the Flink modules in your dependencies section.
-
-If you are looking for building Flink with Scala 2.11, please check
-[build guide]({{ site.baseurl }}/setup/building.html#scala-versions).
-
-#### Hadoop Dependency Versions
-
-If you are using Flink together with Hadoop, the version of the dependency may vary depending on the
-version of Hadoop (or more specifically, HDFS) that you want to use Flink with. Please refer to the
-[downloads page](http://flink.apache.org/downloads.html) for a list of available versions, and instructions
-on how to link with custom versions of Hadoop.
-
-In order to link against the latest SNAPSHOT versions of the code, please follow
-[this guide](http://flink.apache.org/how-to-contribute.html#snapshots-nightly-builds).
-
-The *flink-clients* dependency is only necessary to invoke the Flink program locally (for example to
-run it standalone for testing and debugging). If you intend to only export the program as a JAR
-file and [run it on a cluster]({{ site.baseurl }}/dev/cluster_execution.html), you can skip that dependency.
-
-{% top %}
-
DataSet and DataStream
----------------------
@@ -933,124 +812,6 @@ interface can be implemented by input formats and functions to tell the API
explicitly about their return type. The *input types* that the functions are invoked with can
usually be inferred by the result types of the previous operations.
-Execution Configuration
------------------------
-
-The `StreamExecutionEnvironment` also contains the `ExecutionConfig` which allows to set job specific configuration values for the runtime.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-ExecutionConfig executionConfig = env.getConfig();
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-var executionConfig = env.getConfig
-{% endhighlight %}
-</div>
-</div>
-
-The following configuration options are available: (the default is bold)
-
-- **`enableClosureCleaner()`** / `disableClosureCleaner()`. The closure cleaner is enabled by default. The closure cleaner removes unneeded references to the surrounding class of anonymous functions inside Flink programs.
-With the closure cleaner disabled, it might happen that an anonymous user function is referencing the surrounding class, which is usually not Serializable. This will lead to exceptions by the serializer.
-
-- `getParallelism()` / `setParallelism(int parallelism)` Set the default parallelism for the job.
-
-- `getMaxParallelism()` / `setMaxParallelism(int parallelism)` Set the default maximum parallelism for the job. This setting determines the maximum degree of parallelism and specifies the upper limit for dynamic scaling.
-
-- `getNumberOfExecutionRetries()` / `setNumberOfExecutionRetries(int numberOfExecutionRetries)` Sets the number of times that failed tasks are re-executed. A value of zero effectively disables fault tolerance. A value of `-1` indicates that the system default value (as defined in the configuration) should be used. This is deprecated, use [restart strategies]({{ site.baseurl }}/setup/fault_tolerance.html#restart-strategies) instead.
-
-- `getExecutionRetryDelay()` / `setExecutionRetryDelay(long executionRetryDelay)` Sets the delay in milliseconds that the system waits after a job has failed, before re-executing it. The delay starts after all tasks have been successfully been stopped on the TaskManagers, and once the delay is past, the tasks are re-started. This parameter is useful to delay re-execution in order to let certain time-out related failures surface fully (like broken connections that have not fully timed out), before attempting a re-execution and immediately failing again due to the same problem. This parameter only has an effect if the number of execution re-tries is one or more. This is deprecated, use [restart strategies]({{ site.baseurl }}/setup/fault_tolerance.html#restart-strategies) instead.
-
-- `getExecutionMode()` / `setExecutionMode()`. The default execution mode is PIPELINED. Sets the execution mode to execute the program. The execution mode defines whether data exchanges are performed in a batch or on a pipelined manner.
-
-- `enableForceKryo()` / **`disableForceKryo`**. Kryo is not forced by default. Forces the GenericTypeInformation to use the Kryo serializer for POJOS even though we could analyze them as a POJO. In some cases this might be preferable. For example, when Flink's internal serializers fail to handle a POJO properly.
-
-- `enableForceAvro()` / **`disableForceAvro()`**. Avro is not forced by default. Forces the Flink AvroTypeInformation to use the Avro serializer instead of Kryo for serializing Avro POJOs.
-
-- `enableObjectReuse()` / **`disableObjectReuse()`** By default, objects are not reused in Flink. Enabling the object reuse mode will instruct the runtime to reuse user objects for better performance. Keep in mind that this can lead to bugs when the user-code function of an operation is not aware of this behavior.
-
-- **`enableSysoutLogging()`** / `disableSysoutLogging()` JobManager status updates are printed to `System.out` by default. This setting allows to disable this behavior.
-
-- `getGlobalJobParameters()` / `setGlobalJobParameters()` This method allows users to set custom objects as a global configuration for the job. Since the `ExecutionConfig` is accessible in all user defined functions, this is an easy method for making configuration globally available in a job.
-
-- `addDefaultKryoSerializer(Class<?> type, Serializer<?> serializer)` Register a Kryo serializer instance for the given `type`.
-
-- `addDefaultKryoSerializer(Class<?> type, Class<? extends Serializer<?>> serializerClass)` Register a Kryo serializer class for the given `type`.
-
-- `registerTypeWithKryoSerializer(Class<?> type, Serializer<?> serializer)` Register the given type with Kryo and specify a serializer for it. By registering a type with Kryo, the serialization of the type will be much more efficient.
-
-- `registerKryoType(Class<?> type)` If the type ends up being serialized with Kryo, then it will be registered at Kryo to make sure that only tags (integer IDs) are written. If a type is not registered with Kryo, its entire class-name will be serialized with every instance, leading to much higher I/O costs.
-
-- `registerPojoType(Class<?> type)` Registers the given type with the serialization stack. If the type is eventually serialized as a POJO, then the type is registered with the POJO serializer. If the type ends up being serialized with Kryo, then it will be registered at Kryo to make sure that only tags are written. If a type is not registered with Kryo, its entire class-name will be serialized with every instance, leading to much higher I/O costs.
-
-Note that types registered with `registerKryoType()` are not available to Flink's Kryo serializer instance.
-
-- `disableAutoTypeRegistration()` Automatic type registration is enabled by default. The automatic type registration is registering all types (including sub-types) used by usercode with Kryo and the POJO serializer.
-
-- `setTaskCancellationInterval(long interval)` Sets the the interval (in milliseconds) to wait between consecutive attempts to cancel a running task. When a task is canceled a new thread is created which periodically calls `interrupt()` on the task thread, if the task thread does not terminate within a certain time. This parameter refers to the time between consecutive calls to `interrupt()` and is set by default to **30000** milliseconds, or **30 seconds**.
-
-The `RuntimeContext` which is accessible in `Rich*` functions through the `getRuntimeContext()` method also allows to access the `ExecutionConfig` in all user defined functions.
-
-{% top %}
-
-Program Packaging and Distributed Execution
------------------------------------------
-
-As described earlier, Flink programs can be executed on
-clusters by using a `remote environment`. Alternatively, programs can be packaged into JAR Files
-(Java Archives) for execution. Packaging the program is a prerequisite to executing them through the
-[command line interface]({{ site.baseurl }}/setup/cli.html).
-
-#### Packaging Programs
-
-To support execution from a packaged JAR file via the command line or web interface, a program must
-use the environment obtained by `StreamExecutionEnvironment.getExecutionEnvironment()`. This environment
-will act as the cluster's environment when the JAR is submitted to the command line or web
-interface. If the Flink program is invoked differently than through these interfaces, the
-environment will act like a local environment.
-
-To package the program, simply export all involved classes as a JAR file. The JAR file's manifest
-must point to the class that contains the program's *entry point* (the class with the public
-`main` method). The simplest way to do this is by putting the *main-class* entry into the
-manifest (such as `main-class: org.apache.flinkexample.MyProgram`). The *main-class* attribute is
-the same one that is used by the Java Virtual Machine to find the main method when executing a JAR
-files through the command `java -jar pathToTheJarFile`. Most IDEs offer to include that attribute
-automatically when exporting JAR files.
-
-
-#### Packaging Programs through Plans
-
-Additionally, we support packaging programs as *Plans*. Instead of defining a progam in the main
-method and calling
-`execute()` on the environment, plan packaging returns the *Program Plan*, which is a description of
-the program's data flow. To do that, the program must implement the
-`org.apache.flink.api.common.Program` interface, defining the `getPlan(String...)` method. The
-strings passed to that method are the command line arguments. The program's plan can be created from
-the environment via the `ExecutionEnvironment#createProgramPlan()` method. When packaging the
-program's plan, the JAR manifest must point to the class implementing the
-`org.apache.flinkapi.common.Program` interface, instead of the class with the main method.
-
-
-#### Summary
-
-The overall procedure to invoke a packaged program is as follows:
-
-1. The JAR's manifest is searched for a *main-class* or *program-class* attribute. If both
-attributes are found, the *program-class* attribute takes precedence over the *main-class*
-attribute. Both the command line and the web interface support a parameter to pass the entry point
-class name manually for cases where the JAR manifest contains neither attribute.
-
-2. If the entry point class implements the `org.apache.flinkapi.common.Program`, then the system
-calls the `getPlan(String...)` method to obtain the program plan to execute.
-
-3. If the entry point class does not implement the `org.apache.flinkapi.common.Program` interface,
-the system will invoke the main method of the class.
-
{% top %}
Accumulators & Counters
@@ -1114,7 +875,7 @@ different operator functions of your job. Flink will internally merge all accumu
name.
A note on accumulators and iterations: Currently the result of accumulators is only available after
-the overall job ended. We plan to also make the result of the previous iteration available in the
+the overall job has ended. We plan to also make the result of the previous iteration available in the
next iteration. You can use
{% gh_link /flink-java/src/main/java/org/apache/flink/api/java/operators/IterativeDataSet.java#L98 "Aggregators" %}
to compute per-iteration statistics and base the termination of iterations on such statistics.
@@ -1135,217 +896,3 @@ result type ```R``` for the final result. E.g. for a histogram, ```V``` is a num
{% top %}
-Parallel Execution
-------------------
-
-This section describes how the parallel execution of programs can be configured in Flink. A Flink
-program consists of multiple tasks (transformations/operators, data sources, and sinks). A task is split into
-several parallel instances for execution and each parallel instance processes a subset of the task's
-input data. The number of parallel instances of a task is called its *parallelism*.
-
-
-The parallelism of a task can be specified in Flink on different levels.
-
-### Operator Level
-
-The parallelism of an individual operator, data source, or data sink can be defined by calling its
-`setParallelism()` method. For example, like this:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
-DataStream<String> text = [...]
-DataStream<Tuple2<String, Integer>> wordCounts = text
- .flatMap(new LineSplitter())
- .keyBy(0)
- .timeWindow(Time.seconds(5))
- .sum(1).setParallelism(5);
-
-wordCounts.print();
-
-env.execute("Word Count Example");
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-
-val text = [...]
-val wordCounts = text
- .flatMap{ _.split(" ") map { (_, 1) } }
- .keyBy(0)
- .timeWindow(Time.seconds(5))
- .sum(1).setParallelism(5)
-wordCounts.print()
-
-env.execute("Word Count Example")
-{% endhighlight %}
-</div>
-</div>
-
-### Execution Environment Level
-
-As mentioned [here](#anatomy-of-a-flink-program) Flink programs are executed in the context
-of an execution environment. An
-execution environment defines a default parallelism for all operators, data sources, and data sinks
-it executes. Execution environment parallelism can be overwritten by explicitly configuring the
-parallelism of an operator.
-
-The default parallelism of an execution environment can be specified by calling the
-`setParallelism()` method. To execute all operators, data sources, and data sinks with a parallelism
-of `3`, set the default parallelism of the execution environment as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-env.setParallelism(3);
-
-DataStream<String> text = [...]
-DataStream<Tuple2<String, Integer>> wordCounts = [...]
-wordCounts.print();
-
-env.execute("Word Count Example");
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-env.setParallelism(3)
-
-val text = [...]
-val wordCounts = text
- .flatMap{ _.split(" ") map { (_, 1) } }
- .keyBy(0)
- .timeWindow(Time.seconds(5))
- .sum(1)
-wordCounts.print()
-
-env.execute("Word Count Example")
-{% endhighlight %}
-</div>
-</div>
-
-### Client Level
-
-The parallelism can be set at the Client when submitting jobs to Flink. The
-Client can either be a Java or a Scala program. One example of such a Client is
-Flink's Command-line Interface (CLI).
-
-For the CLI client, the parallelism parameter can be specified with `-p`. For
-example:
-
- ./bin/flink run -p 10 ../examples/*WordCount-java*.jar
-
-
-In a Java/Scala program, the parallelism is set as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-
-try {
- PackagedProgram program = new PackagedProgram(file, args);
- InetSocketAddress jobManagerAddress = RemoteExecutor.getInetFromHostport("localhost:6123");
- Configuration config = new Configuration();
-
- Client client = new Client(jobManagerAddress, config, program.getUserCodeClassLoader());
-
- // set the parallelism to 10 here
- client.run(program, 10, true);
-
-} catch (ProgramInvocationException e) {
- e.printStackTrace();
-}
-
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-try {
- PackagedProgram program = new PackagedProgram(file, args)
- InetSocketAddress jobManagerAddress = RemoteExecutor.getInetFromHostport("localhost:6123")
- Configuration config = new Configuration()
-
- Client client = new Client(jobManagerAddress, new Configuration(), program.getUserCodeClassLoader())
-
- // set the parallelism to 10 here
- client.run(program, 10, true)
-
-} catch {
- case e: Exception => e.printStackTrace
-}
-{% endhighlight %}
-</div>
-</div>
-
-
-### System Level
-
-A system-wide default parallelism for all execution environments can be defined by setting the
-`parallelism.default` property in `./conf/flink-conf.yaml`. See the
-[Configuration]({{ site.baseurl }}/setup/config.html) documentation for details.
-
-{% top %}
-
-Execution Plans
----------------
-
-Depending on various parameters such as data size or number of machines in the cluster, Flink's
-optimizer automatically chooses an execution strategy for your program. In many cases, it can be
-useful to know how exactly Flink will execute your program.
-
-__Plan Visualization Tool__
-
-Flink comes packaged with a visualization tool for execution plans. The HTML document containing
-the visualizer is located under ```tools/planVisualizer.html```. It takes a JSON representation of
-the job execution plan and visualizes it as a graph with complete annotations of execution
-strategies.
-
-The following code shows how to print the execution plan JSON from your program:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-
-...
-
-System.out.println(env.getExecutionPlan());
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-
-...
-
-println(env.getExecutionPlan())
-{% endhighlight %}
-</div>
-</div>
-
-
-To visualize the execution plan, do the following:
-
-1. **Open** ```planVisualizer.html``` with your web browser,
-2. **Paste** the JSON string into the text field, and
-3. **Press** the draw button.
-
-After these steps, a detailed execution plan will be visualized.
-
-<img alt="A flink job execution graph." src="{{ site.baseurl }}/fig/plan_visualizer.png" width="80%">
-
-
-__Web Interface__
-
-Flink offers a web interface for submitting and executing jobs. The interface is part of the JobManager's
-web interface for monitoring, per default running on port 8081. Job submission via this interfaces requires
-that you have set `jobmanager.web.submit.enable: true` in `flink-conf.yaml`.
-
-You may specify program arguments before the job is executed. The plan visualization enables you to show
-the execution plan before executing the Flink job.
-
-{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/apis.md
----------------------------------------------------------------------
diff --git a/docs/dev/apis.md b/docs/dev/apis.md
deleted file mode 100644
index 5e06e14..0000000
--- a/docs/dev/apis.md
+++ /dev/null
@@ -1,24 +0,0 @@
----
-title: "APIs"
-nav-id: apis
-nav-parent_id: dev
-nav-pos: 2
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/batch/examples.md
----------------------------------------------------------------------
diff --git a/docs/dev/batch/examples.md b/docs/dev/batch/examples.md
index 63d6c7a..7b6132c 100644
--- a/docs/dev/batch/examples.md
+++ b/docs/dev/batch/examples.md
@@ -1,8 +1,8 @@
---
-title: "Bundled Examples"
-nav-title: Examples
-nav-parent_id: batch
-nav-pos: 5
+title: "Batch Examples"
+nav-title: Batch Examples
+nav-parent_id: examples
+nav-pos: 20
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -25,7 +25,7 @@ under the License.
The following example programs showcase different applications of Flink
from simple word counting to graph algorithms. The code samples illustrate the
-use of [Flink's API](index.html).
+use of [Flink's DataSet API](/dev/batch/index.html).
The full source code of the following and more examples can be found in the __flink-examples-batch__
or __flink-examples-streaming__ module of the Flink source repository.
@@ -36,7 +36,7 @@ or __flink-examples-streaming__ module of the Flink source repository.
## Running an example
-In order to run a Flink example, we assume you have a running Flink instance available. The "Setup" tab in the navigation describes various ways of starting Flink.
+In order to run a Flink example, we assume you have a running Flink instance available. The "Quickstart" and "Setup" tabs in the navigation describe various ways of starting Flink.
The easiest way is running the `./bin/start-local.sh` script, which will start a JobManager locally.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/batch/index.md
----------------------------------------------------------------------
diff --git a/docs/dev/batch/index.md b/docs/dev/batch/index.md
index 0b1c9f9..48d60e1 100644
--- a/docs/dev/batch/index.md
+++ b/docs/dev/batch/index.md
@@ -2,8 +2,8 @@
title: "Flink DataSet API Programming Guide"
nav-id: batch
nav-title: Batch (DataSet API)
-nav-parent_id: apis
-nav-pos: 3
+nav-parent_id: dev
+nav-pos: 30
nav-show_overview: true
---
<!--
@@ -49,7 +49,7 @@ Example Program
The following program is a complete, working example of WordCount. You can copy & paste the code
to run it locally. You only have to include the correct Flink's library into your project
-(see Section [Linking with Flink]({{ site.baseurl }}/dev/api_concepts.html#linking-with-flink)) and specify the imports. Then you are ready
+(see Section [Linking with Flink]({{ site.baseurl }}/dev/linking_with_flink)) and specify the imports. Then you are ready
to go!
<div class="codetabs" markdown="1">
@@ -272,7 +272,7 @@ DataSet<Tuple3<Integer, String, Double>> output = input.sum(0).andMin(2);
Joins two data sets by creating all pairs of elements that are equal on their keys.
Optionally uses a JoinFunction to turn the pair of elements into a single element, or a
FlatJoinFunction to turn the pair of elements into arbitrarily many (including none)
- elements. See the <a href="#specifying-keys">keys section</a> to learn how to define join keys.
+ elements. See the <a href="/dev/api_concepts#specifying-keys">keys section</a> to learn how to define join keys.
{% highlight java %}
result = input1.join(input2)
.where(0) // key of the first input (tuple field 0)
@@ -298,7 +298,7 @@ result = input1.join(input2, JoinHint.BROADCAST_HASH_FIRST)
<tr>
<td><strong>OuterJoin</strong></td>
<td>
- Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a <code>null</code> value for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the <a href="#specifying-keys">keys section</a> to learn how to define join keys.
+ Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a <code>null</code> value for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the <a href="/dev/api_concepts#specifying-keys">keys section</a> to learn how to define join keys.
{% highlight java %}
input1.leftOuterJoin(input2) // rightOuterJoin or fullOuterJoin for right or full outer joins
.where(0) // key of the first input (tuple field 0)
@@ -320,7 +320,7 @@ input1.leftOuterJoin(input2) // rightOuterJoin or fullOuterJoin for right or ful
<td>
<p>The two-dimensional variant of the reduce operation. Groups each input on one or more
fields and then joins the groups. The transformation function is called per pair of groups.
- See the <a href="#specifying-keys">keys section</a> to learn how to define coGroup keys.</p>
+ See the <a href="/dev/api_concepts#specifying-keys">keys section</a> to learn how to define coGroup keys.</p>
{% highlight java %}
data1.coGroup(data2)
.where(0)
@@ -592,7 +592,7 @@ val output: DataSet[(Int, String, Double)] = input.sum(0).min(2)
Joins two data sets by creating all pairs of elements that are equal on their keys.
Optionally uses a JoinFunction to turn the pair of elements into a single element, or a
FlatJoinFunction to turn the pair of elements into arbitrarily many (including none)
- elements. See the <a href="#specifying-keys">keys section</a> to learn how to define join keys.
+ elements. See the <a href="/dev/api_concepts#specifying-keys">keys section</a> to learn how to define join keys.
{% highlight scala %}
// In this case tuple fields are used as keys. "0" is the join field on the first tuple
// "1" is the join field on the second tuple.
@@ -618,7 +618,7 @@ val result = input1.join(input2, JoinHint.BROADCAST_HASH_FIRST)
<tr>
<td><strong>OuterJoin</strong></td>
<td>
- Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a `null` value for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the <a href="#specifying-keys">keys section</a> to learn how to define join keys.
+ Performs a left, right, or full outer join on two data sets. Outer joins are similar to regular (inner) joins and create all pairs of elements that are equal on their keys. In addition, records of the "outer" side (left, right, or both in case of full) are preserved if no matching key is found in the other side. Matching pairs of elements (or one element and a `null` value for the other input) are given to a JoinFunction to turn the pair of elements into a single element, or to a FlatJoinFunction to turn the pair of elements into arbitrarily many (including none) elements. See the <a href="/dev/api_concepts#specifying-keys">keys section</a> to learn how to define join keys.
{% highlight scala %}
val joined = left.leftOuterJoin(right).where(0).equalTo(1) {
(left, right) =>
@@ -634,7 +634,7 @@ val joined = left.leftOuterJoin(right).where(0).equalTo(1) {
<td>
<p>The two-dimensional variant of the reduce operation. Groups each input on one or more
fields and then joins the groups. The transformation function is called per pair of groups.
- See the <a href="#specifying-keys">keys section</a> to learn how to define coGroup keys.</p>
+ See the <a href="/dev/api_concepts#specifying-keys">keys section</a> to learn how to define coGroup keys.</p>
{% highlight scala %}
data1.coGroup(data2).where(0).equalTo(1)
{% endhighlight %}
@@ -780,7 +780,7 @@ is not supported by the API out-of-the-box. To use this feature, you should use
</div>
</div>
-The [parallelism]({{ site.baseurl }}/dev/api_concepts.html#parallel-execution) of a transformation can be defined by `setParallelism(int)` while
+The [parallelism]({{ site.baseurl }}/dev/parallel) of a transformation can be defined by `setParallelism(int)` while
`name(String)` assigns a custom name to a transformation which is helpful for debugging. The same is
possible for [Data Sources](#data-sources) and [Data Sinks](#data-sinks).
@@ -1990,7 +1990,7 @@ data.map(new RichMapFunction<String, String>() {
Make sure that the names (`broadcastSetName` in the previous example) match when registering and
accessing broadcasted data sets. For a complete example program, have a look at
-{% gh_link /flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java#L96 "K-Means Algorithm" %}.
+{% gh_link /flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java "K-Means Algorithm" %}.
</div>
<div data-lang="scala" markdown="1">
@@ -2016,7 +2016,7 @@ data.map(new RichMapFunction[String, String]() {
Make sure that the names (`broadcastSetName` in the previous example) match when registering and
accessing broadcasted data sets. For a complete example program, have a look at
-{% gh_link /flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala#L96 "KMeans Algorithm" %}.
+{% gh_link /flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala "KMeans Algorithm" %}.
</div>
</div>
@@ -2103,7 +2103,7 @@ val result: DataSet[Integer] = input.map(new MyMapper())
env.execute()
{% endhighlight %}
-Access the cached file in a user function (here a `MapFunction`). The function must extend a [RichFunction]({{ site.baseurl }}/apis/common/index.html#rich-functions) class because it needs access to the `RuntimeContext`.
+Access the cached file in a user function (here a `MapFunction`). The function must extend a [RichFunction]({{ site.baseurl }}/dev/api_concepts#rich-functions) class because it needs access to the `RuntimeContext`.
{% highlight scala %}
@@ -2258,7 +2258,7 @@ Please note that you can also pass a custom class extending the `ExecutionConfig
**Accessing values from the global configuration**
-Objects in the global job parameters are accessible in many places in the system. All user functions implementing a `Rich*Function` interface have access through the runtime context.
+Objects in the global job parameters are accessible in many places in the system. All user functions implementing a `RichFunction` interface have access through the runtime context.
{% highlight java %}
public static final class Tokenizer extends RichFlatMapFunction<String, Tuple2<String, Integer>> {
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/batch/iterations.md
----------------------------------------------------------------------
diff --git a/docs/dev/batch/iterations.md b/docs/dev/batch/iterations.md
index cfeffe6..73a1d57 100644
--- a/docs/dev/batch/iterations.md
+++ b/docs/dev/batch/iterations.md
@@ -1,9 +1,7 @@
---
-title: "Iterations"
-
-# Sub-level navigation
-sub-nav-group: batch
-sub-nav-pos: 3
+title: Iterations
+nav-parent_id: batch
+nav-pos: 2
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/cluster_execution.md
----------------------------------------------------------------------
diff --git a/docs/dev/cluster_execution.md b/docs/dev/cluster_execution.md
index 31b4d4a..d614846 100644
--- a/docs/dev/cluster_execution.md
+++ b/docs/dev/cluster_execution.md
@@ -1,6 +1,6 @@
---
title: "Cluster Execution"
-nav-parent_id: dev
+nav-parent_id: batch
nav-pos: 12
---
<!--
@@ -81,75 +81,3 @@ public static void main(String[] args) throws Exception {
Note that the program contains custom user code and hence requires a JAR file with
the classes of the code attached. The constructor of the remote environment
takes the path(s) to the JAR file(s).
-
-## Linking with modules not contained in the binary distribution
-
-The binary distribution contains jar packages in the `lib` folder that are automatically
-provided to the classpath of your distributed programs. Almost all of Flink classes are
-located there with a few exceptions, for example the streaming connectors and some freshly
-added modules. To run code depending on these modules you need to make them accessible
-during runtime, for which we suggest two options:
-
-1. Either copy the required jar files to the `lib` folder onto all of your TaskManagers.
-Note that you have to restart your TaskManagers after this.
-2. Or package them with your code.
-
-The latter version is recommended as it respects the classloader management in Flink.
-
-### Packaging dependencies with your usercode with Maven
-
-To provide these dependencies not included by Flink we suggest two options with Maven.
-
-1. The maven assembly plugin builds a so-called uber-jar (executable jar) containing all your dependencies.
-The assembly configuration is straight-forward, but the resulting jar might become bulky.
-See [maven-assembly-plugin](http://maven.apache.org/plugins/maven-assembly-plugin/usage.html) for further information.
-2. The maven unpack plugin unpacks the relevant parts of the dependencies and
-then packages it with your code.
-
-Using the latter approach in order to bundle the Kafka connector, `flink-connector-kafka`
-you would need to add the classes from both the connector and the Kafka API itself. Add
-the following to your plugins section.
-
-~~~xml
-<plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-dependency-plugin</artifactId>
- <version>2.9</version>
- <executions>
- <execution>
- <id>unpack</id>
- <!-- executed just before the package phase -->
- <phase>prepare-package</phase>
- <goals>
- <goal>unpack</goal>
- </goals>
- <configuration>
- <artifactItems>
- <!-- For Flink connector classes -->
- <artifactItem>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-kafka</artifactId>
- <version>{{ site.version }}</version>
- <type>jar</type>
- <overWrite>false</overWrite>
- <outputDirectory>${project.build.directory}/classes</outputDirectory>
- <includes>org/apache/flink/**</includes>
- </artifactItem>
- <!-- For Kafka API classes -->
- <artifactItem>
- <groupId>org.apache.kafka</groupId>
- <artifactId>kafka_<YOUR_SCALA_VERSION></artifactId>
- <version><YOUR_KAFKA_VERSION></version>
- <type>jar</type>
- <overWrite>false</overWrite>
- <outputDirectory>${project.build.directory}/classes</outputDirectory>
- <includes>kafka/**</includes>
- </artifactItem>
- </artifactItems>
- </configuration>
- </execution>
- </executions>
-</plugin>
-~~~
-
-Now when running `mvn clean package` the produced jar includes the required dependencies.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/cassandra.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/cassandra.md b/docs/dev/connectors/cassandra.md
index 90be0e3..19d483b 100644
--- a/docs/dev/connectors/cassandra.md
+++ b/docs/dev/connectors/cassandra.md
@@ -35,7 +35,7 @@ To use this connector, add the following dependency to your project:
</dependency>
{% endhighlight %}
-Note that the streaming connectors are currently not part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
+Note that the streaming connectors are currently not part of the binary distribution. See how to link with them for cluster execution [here]({{ site.baseurl}}/dev/linking).
#### Installing Apache Cassandra
Follow the instructions from the [Cassandra Getting Started page](http://wiki.apache.org/cassandra/GettingStarted).
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/elasticsearch.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/elasticsearch.md b/docs/dev/connectors/elasticsearch.md
index be45f98..1907740 100644
--- a/docs/dev/connectors/elasticsearch.md
+++ b/docs/dev/connectors/elasticsearch.md
@@ -37,7 +37,7 @@ following dependency to your project:
Note that the streaming connectors are currently not part of the binary
distribution. See
-[here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution)
+[here]({{site.baseurl}}/dev/linking)
for information about how to package the program with the libraries for
cluster execution.
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/elasticsearch2.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/elasticsearch2.md b/docs/dev/connectors/elasticsearch2.md
index 5f4267e..a796280 100644
--- a/docs/dev/connectors/elasticsearch2.md
+++ b/docs/dev/connectors/elasticsearch2.md
@@ -37,7 +37,7 @@ following dependency to your project:
Note that the streaming connectors are currently not part of the binary
distribution. See
-[here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution)
+[here]({{site.baseurl}}/dev/linking)
for information about how to package the program with the libraries for
cluster execution.
@@ -145,7 +145,7 @@ More information about Elasticsearch can be found [here](https://elastic.co).
For the execution of your Flink program,
it is recommended to build a so-called uber-jar (executable jar) containing all your dependencies
-(see [here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution) for further information).
+(see [here]({{site.baseurl}}/dev/linking) for further information).
However,
when an uber-jar containing an Elasticsearch sink is executed,
http://git-wip-us.apache.org/repos/asf/flink/blob/79d7e301/docs/dev/connectors/filesystem_sink.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/filesystem_sink.md b/docs/dev/connectors/filesystem_sink.md
index 79cfe6a..030e9d9 100644
--- a/docs/dev/connectors/filesystem_sink.md
+++ b/docs/dev/connectors/filesystem_sink.md
@@ -37,7 +37,7 @@ following dependency to your project:
Note that the streaming connectors are currently not part of the binary
distribution. See
-[here]({{site.baseurl}}/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution)
+[here]({{site.baseurl}}/dev/linking)
for information about how to package the program with the libraries for
cluster execution.