You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by uc...@apache.org on 2015/04/22 16:17:00 UTC
[03/30] flink git commit: [docs] Change doc layout
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/config.md
----------------------------------------------------------------------
diff --git a/docs/setup/config.md b/docs/setup/config.md
new file mode 100644
index 0000000..a48c6d7
--- /dev/null
+++ b/docs/setup/config.md
@@ -0,0 +1,385 @@
+---
+title: "Configuration"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Overview
+
+The default configuration parameters allow Flink to run out-of-the-box
+in single node setups.
+
+This page lists the most common options that are typically needed to set
+up a well performing (distributed) installation. In addition a full
+list of all available configuration parameters is listed here.
+
+All configuration is done in `conf/flink-conf.yaml`, which is expected to be
+a flat collection of [YAML key value pairs](http://www.yaml.org/spec/1.2/spec.html)
+with format `key: value`.
+
+The system and run scripts parse the config at startup time. Changes to the configuration
+file require restarting the Flink JobManager and TaskManagers.
+
+The configuration files for the TaskManagers can be different, Flink does not assume
+uniform machines in the cluster.
+
+* This will be replaced by the TOC
+{:toc}
+
+
+## Common Options
+
+- `env.java.home`: The path to the Java installation to use (DEFAULT: system's
+default Java installation, if found). Needs to be specified if the startup
+scipts fail to automatically resolve the java home directory. Can be specified
+to point to a specific java installation or version. If this option is not
+specified, the startup scripts also evaluate the `$JAVA_HOME` environment variable.
+
+- `jobmanager.rpc.address`: The IP address of the JobManager, which is the
+master/coordinator of the distributed system (DEFAULT: localhost).
+
+- `jobmanager.rpc.port`: The port number of the JobManager (DEFAULT: 6123).
+
+- `jobmanager.heap.mb`: JVM heap size (in megabytes) for the JobManager. You may have to increase the heap size for the JobManager if you are running
+very large applications (with many operators), or if you are keeping a long history of them.
+
+- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers,
+which are the parallel workers of the system. In
+contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and
+user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager
+(including sorting/hashing/caching), so this value should be as
+large as possible. If the cluster is exclusively running Flink,
+the total amount of available memory per machine minus some memory for the
+operating system (maybe 1-2 GB) is a good value.
+On YARN setups, this value is automatically configured to the size of
+the TaskManager's YARN container, minus a certain tolerance value.
+
+- `taskmanager.numberOfTaskSlots`: The number of parallel operator or
+user function instances that a single TaskManager can run (DEFAULT: 1).
+If this value is larger than 1, a single TaskManager takes multiple instances of
+a function or operator. That way, the TaskManager can utilize multiple CPU cores,
+but at the same time, the available memory is divided between the different
+operator or function instances.
+This value is typically proportional to the number of physical CPU cores that
+the TaskManager's machine has (e.g., equal to the number of cores, or half the
+number of cores). [More about task slots](config.html#configuring-taskmanager-processing-slots).
+
+- `parallelism.default`: The default parallelism to use for programs that have
+no parallelism specified. (DEFAULT: 1). For setups that have no concurrent jobs
+running, setting this value to NumTaskManagers * NumSlotsPerTaskManager will
+cause the system to use all available execution resources for the program's
+execution. **Note**: The default parallelism can be overwriten for an entire
+job by calling `setParallelism(int parallelism)` on the `ExecutionEnvironment`
+or by passing `-p <parallelism>` to the Flink Command-line frontend. It can be
+overwritten for single transformations by calling `setParallelism(int
+parallelism)` on an operator. See the [programming
+guide](programming_guide.html#parallel-execution) for more information about the
+parallelism.
+
+- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop File System's (HDFS)
+configuration directory (OPTIONAL VALUE).
+Specifying this value allows programs to reference HDFS files using short URIs
+(`hdfs:///path/to/files`, without including the address and port of the NameNode
+in the file URI). Without this option, HDFS files can be accessed, but require
+fully qualified URIs like `hdfs://address:port/path/to/files`.
+This option also causes file writers to pick up the HDFS's default values for block sizes
+and replication factors. Flink will look for the "core-site.xml" and
+"hdfs-site.xml" files in teh specified directory.
+
+
+## Advanced Options
+
+- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of
+directories separated by the systems directory delimiter (for example ':'
+(colon) on Linux/Unix). If multiple directories are specified, then the temporary
+files will be distributed across the directories in a round-robin fashion. The
+I/O manager component will spawn one reading and one writing thread per
+directory. A directory may be listed multiple times to have the I/O manager use
+multiple threads for it (for example if it is physically stored on a very fast
+disc or RAID) (DEFAULT: The system's tmp dir).
+
+- `jobmanager.web.port`: Port of the JobManager's web interface (DEFAULT: 8081).
+
+- `fs.overwrite-files`: Specifies whether file output writers should overwrite
+existing files by default. Set to *true* to overwrite by default, *false* otherwise.
+(DEFAULT: false)
+
+- `fs.output.always-create-directory`: File writers running with a parallelism
+larger than one create a directory for the output file path and put the different
+result files (one per parallel writer task) into that directory. If this option
+is set to *true*, writers with a parallelism of 1 will also create a directory
+and place a single result file into it. If the option is set to *false*, the
+writer will directly create the file directly at the output path, without
+creating a containing directory. (DEFAULT: false)
+
+- `taskmanager.network.numberOfBuffers`: The number of buffers available to the
+network stack. This number determines how many streaming data exchange channels
+a TaskManager can have at the same time and how well buffered the channels are.
+If a job is rejected or you get a warning that the system has not enough buffers
+available, increase this value (DEFAULT: 2048).
+
+- `taskmanager.memory.size`: The amount of memory (in megabytes) that the task
+manager reserves on the JVM's heap space for sorting, hash tables, and caching
+of intermediate results. If unspecified (-1), the memory manager will take a fixed
+ratio of the heap memory available to the JVM, as specified by
+`taskmanager.memory.fraction`. (DEFAULT: -1)
+
+- `taskmanager.memory.fraction`: The relative amount of memory that the task
+manager reserves for sorting, hash tables, and caching of intermediate results.
+For example, a value of 0.8 means that TaskManagers reserve 80% of the
+JVM's heap space for internal data buffers, leaving 20% of the JVM's heap space
+free for objects created by user-defined functions. (DEFAULT: 0.7)
+This parameter is only evaluated, if `taskmanager.memory.size` is not set.
+
+
+## Full Reference
+
+### HDFS
+
+These parameters configure the default HDFS used by Flink. Setups that do not
+specify a HDFS configuration have to specify the full path to
+HDFS files (`hdfs://address:port/path/to/files`) Files will also be written
+with default HDFS parameters (block size, replication factor).
+
+- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop configuration directory.
+The system will look for the "core-site.xml" and "hdfs-site.xml" files in that
+directory (DEFAULT: null).
+- `fs.hdfs.hdfsdefault`: The absolute path of Hadoop's own configuration file
+"hdfs-default.xml" (DEFAULT: null).
+- `fs.hdfs.hdfssite`: The absolute path of Hadoop's own configuration file
+"hdfs-site.xml" (DEFAULT: null).
+
+### JobManager & TaskManager
+
+The following parameters configure Flink's JobManager and TaskManagers.
+
+- `jobmanager.rpc.address`: The IP address of the JobManager, which is the
+master/coordinator of the distributed system (DEFAULT: localhost).
+- `jobmanager.rpc.port`: The port number of the JobManager (DEFAULT: 6123).
+- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: 6122).
+- `taskmanager.data.port`: The task manager's port used for data exchange
+operations (DEFAULT: 6121).
+- `jobmanager.heap.mb`: JVM heap size (in megabytes) for the JobManager
+(DEFAULT: 256).
+- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers,
+which are the parallel workers of the system. In
+contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and
+user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager
+(including sorting/hashing/caching), so this value should be as
+large as possible (DEFAULT: 512). On YARN setups, this value is automatically
+configured to the size of the TaskManager's YARN container, minus a
+certain tolerance value.
+- `taskmanager.numberOfTaskSlots`: The number of parallel operator or
+user function instances that a single TaskManager can run (DEFAULT: 1).
+If this value is larger than 1, a single TaskManager takes multiple instances of
+a function or operator. That way, the TaskManager can utilize multiple CPU cores,
+but at the same time, the available memory is divided between the different
+operator or function instances.
+This value is typically proportional to the number of physical CPU cores that
+the TaskManager's machine has (e.g., equal to the number of cores, or half the
+number of cores).
+- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of
+directories separated by the systems directory delimiter (for example ':'
+(colon) on Linux/Unix). If multiple directories are specified, then the temporary
+files will be distributed across the directories in a round robin fashion. The
+I/O manager component will spawn one reading and one writing thread per
+directory. A directory may be listed multiple times to have the I/O manager use
+multiple threads for it (for example if it is physically stored on a very fast
+disc or RAID) (DEFAULT: The system's tmp dir).
+- `taskmanager.network.numberOfBuffers`: The number of buffers available to the
+network stack. This number determines how many streaming data exchange channels
+a TaskManager can have at the same time and how well buffered the channels are.
+If a job is rejected or you get a warning that the system has not enough buffers
+available, increase this value (DEFAULT: 2048).
+- `taskmanager.network.bufferSizeInBytes`: The size of the network buffers, in
+bytes (DEFAULT: 32768 (= 32 KiBytes)).
+- `taskmanager.memory.size`: The amount of memory (in megabytes) that the task
+manager reserves on the JVM's heap space for sorting, hash tables, and caching
+of intermediate results. If unspecified (-1), the memory manager will take a fixed
+ratio of the heap memory available to the JVM, as specified by
+`taskmanager.memory.fraction`. (DEFAULT: -1)
+- `taskmanager.memory.fraction`: The relative amount of memory that the task
+manager reserves for sorting, hash tables, and caching of intermediate results.
+For example, a value of 0.8 means that TaskManagers reserve 80% of the
+JVM's heap space for internal data buffers, leaving 20% of the JVM's heap space
+free for objects created by user-defined functions. (DEFAULT: 0.7)
+This parameter is only evaluated, if `taskmanager.memory.size` is not set.
+- `jobclient.polling.interval`: The interval (in seconds) in which the client
+polls the JobManager for the status of its job (DEFAULT: 2).
+- `taskmanager.runtime.max-fan`: The maximal fan-in for external merge joins and
+fan-out for spilling hash tables. Limits the number of file handles per operator,
+but may cause intermediate merging/partitioning, if set too small (DEFAULT: 128).
+- `taskmanager.runtime.sort-spilling-threshold`: A sort operation starts spilling
+when this fraction of its memory budget is full (DEFAULT: 0.8).
+- `taskmanager.heartbeat-interval`: The interval in which the TaskManager sends
+heartbeats to the JobManager.
+- `jobmanager.max-heartbeat-delay-before-failure.msecs`: The maximum time that a
+TaskManager hearbeat may be missing before the TaskManager is considered failed.
+
+### Distributed Coordination (via Akka)
+
+- `akka.ask.timeout`: Timeout used for all futures and blocking Akka calls. If Flink fails due to timeouts then you should try to increase this value. Timeouts can be caused by slow machines or a congested network. The timeout value requires a time-unit specifier (ms/s/min/h/d) (DEFAULT: **100 s**).
+- `akka.lookup.timeout`: Timeout used for the lookup of the JobManager. The timeout value has to contain a time-unit specifier (ms/s/min/h/d) (DEFAULT: **10 s**).
+- `akka.framesize`: Maximum size of messages which are sent between the JobManager and the TaskManagers. If Flink fails because messages exceed this limit, then you should increase it. The message size requires a size-unit specifier (DEFAULT: **10485760b**).
+- `akka.watch.heartbeat.interval`: Heartbeat interval for Akka's DeathWatch mechanism to detect dead TaskManagers. If TaskManagers are wrongly marked dead because of lost or delayed heartbeat messages, then you should increase this value. A thorough description of Akka's DeathWatch can be found [here](http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector) (DEFAULT: **akka.ask.timeout/10**).
+- `akka.watch.heartbeat.pause`: Acceptable heartbeat pause for Akka's DeathWatch mechanism. A low value does not allow a irregular heartbeat. A thorough description of Akka's DeathWatch can be found [here](http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector) (DEFAULT: **akka.ask.timeout**).
+- `akka.watch.threshold`: Threshold for the DeathWatch failure detector. A low value is prone to false positives whereas a high value increases the time to detect a dead TaskManager. A thorough description of Akka's DeathWatch can be found [here](http://doc.akka.io/docs/akka/snapshot/scala/remoting.html#failure-detector) (DEFAULT: **12**).
+- `akka.transport.heartbeat.interval`: Heartbeat interval for Akka's transport failure detector. Since Flink uses TCP, the detector is not necessary. Therefore, the detector is disabled by setting the interval to a very high value. In case you should need the transport failure detector, set the interval to some reasonable value. The interval value requires a time-unit specifier (ms/s/min/h/d) (DEFAULT: **1000 s**).
+- `akka.transport.heartbeat.pause`: Acceptable heartbeat pause for Akka's transport failure detector. Since Flink uses TCP, the detector is not necessary. Therefore, the detector is disabled by setting the pause to a very high value. In case you should need the transport failure detector, set the pause to some reasonable value. The pause value requires a time-unit specifier (ms/s/min/h/d) (DEFAULT: **6000 s**).
+- `akka.transport.threshold`: Threshold for the transport failure detector. Since Flink uses TCP, the detector is not necessary and, thus, the threshold is set to a high value (DEFAULT: **300**).
+- `akka.tcp.timeout`: Timeout for all outbound connections. If you should experience problems with connecting to a TaskManager due to a slow network, you should increase this value (DEFAULT: **akka.ask.timeout**).
+- `akka.throughput`: Number of messages that are processed in a batch before returning the thread to the pool. Low values denote a fair scheduling whereas high values can increase the performance at the cost of unfairness (DEFAULT: **15**).
+- `akka.log.lifecycle.events`: Turns on the Akka's remote logging of events. Set this value to 'on' in case of debugging (DEFAULT: **off**).
+- `akka.startup-timeout`: Timeout after which the startup of a remote component is considered being failed (DEFAULT: **akka.ask.timeout**).
+
+### JobManager Web Frontend
+
+- `jobmanager.web.port`: Port of the JobManager's web interface that displays
+status of running jobs and execution time breakdowns of finished jobs
+(DEFAULT: 8081). Setting this value to `-1` disables the web frontend.
+- `jobmanager.web.history`: The number of latest jobs that the JobManager's web
+front-end in its history (DEFAULT: 5).
+
+### Webclient
+
+These parameters configure the web interface that can be used to submit jobs and
+review the compiler's execution plans.
+
+- `webclient.port`: The port of the webclient server (DEFAULT: 8080).
+- `webclient.tempdir`: The temp directory for the web server. Used for example
+for caching file fragments during file-uploads (DEFAULT: The system's temp
+directory).
+- `webclient.uploaddir`: The directory into which the web server will store
+uploaded programs (DEFAULT: ${webclient.tempdir}/webclient-jobs/).
+- `webclient.plandump`: The directory into which the web server will dump
+temporary JSON files describing the execution plans
+(DEFAULT: ${webclient.tempdir}/webclient-plans/).
+
+### File Systems
+
+The parameters define the behavior of tasks that create result files.
+
+- `fs.overwrite-files`: Specifies whether file output writers should overwrite
+existing files by default. Set to *true* to overwrite by default, *false* otherwise.
+(DEFAULT: false)
+- `fs.output.always-create-directory`: File writers running with a parallelism
+larger than one create a directory for the output file path and put the different
+result files (one per parallel writer task) into that directory. If this option
+is set to *true*, writers with a parallelism of 1 will also create a directory
+and place a single result file into it. If the option is set to *false*, the
+writer will directly create the file directly at the output path, without
+creating a containing directory. (DEFAULT: false)
+
+### Compiler/Optimizer
+
+- `compiler.delimited-informat.max-line-samples`: The maximum number of line
+samples taken by the compiler for delimited inputs. The samples are used to
+estimate the number of records. This value can be overridden for a specific
+input with the input format's parameters (DEFAULT: 10).
+- `compiler.delimited-informat.min-line-samples`: The minimum number of line
+samples taken by the compiler for delimited inputs. The samples are used to
+estimate the number of records. This value can be overridden for a specific
+input with the input format's parameters (DEFAULT: 2).
+- `compiler.delimited-informat.max-sample-len`: The maximal length of a line
+sample that the compiler takes for delimited inputs. If the length of a single
+sample exceeds this value (possible because of misconfiguration of the parser),
+the sampling aborts. This value can be overridden for a specific input with the
+input format's parameters (DEFAULT: 2097152 (= 2 MiBytes)).
+
+## YARN
+
+Please note that all ports used by Flink in a YARN session are offsetted by the YARN application ID
+to avoid duplicate port allocations when running multiple YARN sessions in parallel.
+
+So if `yarn.am.rpc.port` is configured to `10245` and the session's application ID is `application_1406629969999_0002`, then the actual port being used is 10245 + 2 = 10247
+
+- `yarn.heap-cutoff-ratio`: Percentage of heap space to remove from containers started by YARN.
+
+
+## Background
+
+### Configuring the Network Buffers
+
+Network buffers are a critical resource for the communication layers. They are
+used to buffer records before transmission over a network, and to buffer
+incoming data before dissecting it into records and handing them to the
+application. A sufficient number of network buffers is critical to achieve a
+good throughput.
+
+In general, configure the task manager to have enough buffers that each logical
+network connection on you expect to be open at the same time has a dedicated
+buffer. A logical network connection exists for each point-to-point exchange of
+data over the network, which typically happens at repartitioning- or
+broadcasting steps. In those, each parallel task inside the TaskManager has to
+be able to talk to all other parallel tasks. Hence, the required number of
+buffers on a task manager is *total-degree-of-parallelism* (number of targets)
+\* *intra-node-parallelism* (number of sources in one task manager) \* *n*.
+Here, *n* is a constant that defines how many repartitioning-/broadcasting steps
+you expect to be active at the same time.
+
+Since the *intra-node-parallelism* is typically the number of cores, and more
+than 4 repartitioning or broadcasting channels are rarely active in parallel, it
+frequently boils down to *\#cores\^2\^* \* *\#machines* \* 4. To support for
+example a cluster of 20 8-core machines, you should use roughly 5000 network
+buffers for optimal throughput.
+
+Each network buffer has by default a size of 32 KiBytes. In the above example, the
+system would allocate roughly 300 MiBytes for network buffers.
+
+The number and size of network buffers can be configured with the following
+parameters:
+
+- `taskmanager.network.numberOfBuffers`, and
+- `taskmanager.network.bufferSizeInBytes`.
+
+### Configuring Temporary I/O Directories
+
+Although Flink aims to process as much data in main memory as possible,
+it is not uncommon that more data needs to be processed than memory is
+available. Flink's runtime is designed to write temporary data to disk
+to handle these situations.
+
+The `taskmanager.tmp.dirs` parameter specifies a list of directories into which
+Flink writes temporary files. The paths of the directories need to be
+separated by ':' (colon character). Flink will concurrently write (or
+read) one temporary file to (from) each configured directory. This way,
+temporary I/O can be evenly distributed over multiple independent I/O devices
+such as hard disks to improve performance. To leverage fast I/O devices (e.g.,
+SSD, RAID, NAS), it is possible to specify a directory multiple times.
+
+If the `taskmanager.tmp.dirs` parameter is not explicitly specified,
+Flink writes temporary data to the temporary directory of the operating
+system, such as */tmp* in Linux systems.
+
+
+### Configuring TaskManager processing slots
+
+Flink executes a program in parallel by splitting it into subtasks and scheduling these subtasks to processing slots.
+
+Each Flink TaskManager provides processing slots in the cluster. The number of slots
+is typically proportional to the number of available CPU cores __of each__ TaskManager.
+As a general recommendation, the number of available CPU cores is a good default for
+`taskmanager.numberOfTaskSlots`.
+
+When starting a Flink application, users can supply the default number of slots to use for that job.
+The command line value therefore is called `-p` (for parallelism). In addition, it is possible
+to [set the number of slots in the programming APIs](programming_guide.html#parallel-execution) for
+the whole application and individual operators.
+
+<img src="img/slots_parallelism.svg" class="img-responsive" />
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/fig/FlinkOnYarn.svg
----------------------------------------------------------------------
diff --git a/docs/setup/fig/FlinkOnYarn.svg b/docs/setup/fig/FlinkOnYarn.svg
new file mode 100644
index 0000000..3eddf50
--- /dev/null
+++ b/docs/setup/fig/FlinkOnYarn.svg
@@ -0,0 +1,151 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" fill-opacity="1" color-rendering="auto" color-interpolation="auto" stroke="black" text-rendering="auto" stroke-linecap="square" width="877" stroke-miterlimit="10" stroke-opacity="1" shape-rendering="auto" fill="black" stroke-dasharray="none" font-weight="normal" stroke-width="1" height="397" font-family="'Dialog'" font-style="normal" stroke-linejoin="miter" font-size="12" stroke-dashoffset="0" image-rendering="auto">
+ <!--Generated by ySVG 2.5-->
+ <defs id="genericDefs"/>
+ <g>
+ <defs id="defs1">
+ <linearGradient x1="336.5" gradientUnits="userSpaceOnUse" x2="506.5" y1="386.5" y2="426.5" id="linearGradient1" spreadMethod="reflect">
+ <stop stop-opacity="1" stop-color="rgb(232,238,247)" offset="0%"/>
+ <stop stop-opacity="1" stop-color="rgb(183,201,227)" offset="100%"/>
+ </linearGradient>
+ <clipPath clipPathUnits="userSpaceOnUse" id="clipPath1">
+ <path d="M0 0 L877 0 L877 397 L0 397 L0 0 Z"/>
+ </clipPath>
+ <clipPath clipPathUnits="userSpaceOnUse" id="clipPath2">
+ <path d="M77 45 L954 45 L954 442 L77 442 L77 45 Z"/>
+ </clipPath>
+ </defs>
+ <g fill="white" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="translate(-77,-45)" stroke="white">
+ <rect x="77" width="877" height="397" y="45" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g fill="rgb(0,153,153)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(0,153,153)">
+ <rect x="92" y="209" clip-path="url(#clipPath2)" width="119" rx="4" ry="4" height="117" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="107.9238" xml:space="preserve" y="226.1387" clip-path="url(#clipPath2)" stroke="none">"Master" Node</text>
+ <rect x="92" y="209" clip-path="url(#clipPath2)" fill="none" width="119" rx="4" ry="4" height="117"/>
+ </g>
+ <g fill="rgb(0,153,153)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(0,153,153)">
+ <rect x="369" y="60" clip-path="url(#clipPath2)" width="105" rx="4" ry="4" height="117" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="375.498" xml:space="preserve" y="115.6699" clip-path="url(#clipPath2)" stroke="none">YARN Resource</text>
+ <text x="395.4492" xml:space="preserve" y="129.6387" clip-path="url(#clipPath2)" stroke="none">Manager</text>
+ <rect x="369" y="60" clip-path="url(#clipPath2)" fill="none" width="105" rx="4" ry="4" height="117"/>
+ </g>
+ <g fill="rgb(0,153,153)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(0,153,153)">
+ <rect x="369" y="212" clip-path="url(#clipPath2)" width="105" rx="4" ry="4" height="117" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="373.8457" xml:space="preserve" y="229.1387" clip-path="url(#clipPath2)" stroke="none">YARN Container</text>
+ <rect x="369" y="212" clip-path="url(#clipPath2)" fill="none" width="105" rx="4" ry="4" height="117"/>
+ </g>
+ <g fill="rgb(0,153,153)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(0,153,153)">
+ <rect x="614" y="212" clip-path="url(#clipPath2)" width="105" rx="4" ry="4" height="117" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="618.8457" xml:space="preserve" y="229.1387" clip-path="url(#clipPath2)" stroke="none">YARN Container</text>
+ <rect x="614" y="212" clip-path="url(#clipPath2)" fill="none" width="105" rx="4" ry="4" height="117"/>
+ </g>
+ <g fill="url(#linearGradient1)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="url(#linearGradient1)">
+ <path d="M336.5 394.5 C341.6 386.5 501.4 386.5 506.5 394.5 L506.5 418.5 C501.4 426.5 341.6 426.5 336.5 418.5 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <path fill="none" d="M336.5 394.5 C341.6 386.5 501.4 386.5 506.5 394.5 L506.5 418.5 C501.4 426.5 341.6 426.5 336.5 418.5 Z" clip-path="url(#clipPath2)"/>
+ <path fill="none" d="M506.5 394.5 C501.4 402.5 341.6 402.5 336.5 394.5" clip-path="url(#clipPath2)"/>
+ <text x="405.1084" xml:space="preserve" y="410.6543" font-family="sans-serif" clip-path="url(#clipPath2)" stroke="none">HDFS</text>
+ </g>
+ <g fill="rgb(0,153,153)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(0,153,153)">
+ <rect x="791" y="212" clip-path="url(#clipPath2)" width="105" rx="4" ry="4" height="117" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="795.8457" xml:space="preserve" y="229.1387" clip-path="url(#clipPath2)" stroke="none">YARN Container</text>
+ <rect x="791" y="212" clip-path="url(#clipPath2)" fill="none" width="105" rx="4" ry="4" height="117"/>
+ </g>
+ <g fill="rgb(51,153,102)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(51,153,102)">
+ <rect x="99" width="105" height="30" y="242.5" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="137.4375" xml:space="preserve" y="254.6699" clip-path="url(#clipPath2)" stroke="none">Flink</text>
+ <text x="115.7959" xml:space="preserve" y="268.6387" clip-path="url(#clipPath2)" stroke="none">YARN Client</text>
+ <rect fill="none" x="99" width="105" height="30" y="242.5" clip-path="url(#clipPath2)"/>
+ </g>
+ <g fill="rgb(51,153,102)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(51,153,102)">
+ <rect x="375" width="93" height="30" y="252.5" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="407.4375" xml:space="preserve" y="264.6699" clip-path="url(#clipPath2)" stroke="none">Flink</text>
+ <text x="385.9512" xml:space="preserve" y="278.6387" clip-path="url(#clipPath2)" stroke="none">JobManager</text>
+ <rect fill="none" x="375" width="93" height="30" y="252.5" clip-path="url(#clipPath2)"/>
+ </g>
+ <g fill="rgb(51,153,102)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(51,153,102)">
+ <rect x="375" width="93" height="30" y="282.5" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="389.5371" xml:space="preserve" y="294.6699" clip-path="url(#clipPath2)" stroke="none">YARN App.</text>
+ <text x="401.0098" xml:space="preserve" y="308.6387" clip-path="url(#clipPath2)" stroke="none">Master</text>
+ <rect fill="none" x="375" width="93" height="30" y="282.5" clip-path="url(#clipPath2)"/>
+ </g>
+ <g fill="rgb(51,153,102)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(51,153,102)">
+ <rect x="620" width="93" height="30" y="255.5" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="652.4375" xml:space="preserve" y="267.6699" clip-path="url(#clipPath2)" stroke="none">Flink</text>
+ <text x="626.2578" xml:space="preserve" y="281.6387" clip-path="url(#clipPath2)" stroke="none">TaskManager</text>
+ <rect fill="none" x="620" width="93" height="30" y="255.5" clip-path="url(#clipPath2)"/>
+ </g>
+ <g fill="rgb(51,153,102)" text-rendering="geometricPrecision" shape-rendering="geometricPrecision" transform="matrix(1,0,0,1,-77,-45)" stroke="rgb(51,153,102)">
+ <rect x="797" width="93" height="30" y="255.5" clip-path="url(#clipPath2)" stroke="none"/>
+ </g>
+ <g text-rendering="geometricPrecision" stroke-miterlimit="1.45" shape-rendering="geometricPrecision" font-family="sans-serif" transform="matrix(1,0,0,1,-77,-45)" stroke-linecap="butt">
+ <text x="829.4375" xml:space="preserve" y="267.6699" clip-path="url(#clipPath2)" stroke="none">Flink</text>
+ <text x="803.2578" xml:space="preserve" y="281.6387" clip-path="url(#clipPath2)" stroke="none">TaskManager</text>
+ <rect fill="none" x="797" width="93" height="30" y="255.5" clip-path="url(#clipPath2)"/>
+ <text x="917.1621" xml:space="preserve" y="274.6543" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">...</text>
+ <path fill="none" d="M210.991 234.6698 L361.997 151.3369" clip-path="url(#clipPath2)"/>
+ <path d="M369.0012 147.4715 L356.079 148.8919 L361.1215 151.82 L360.9107 157.6472 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <text x="157.7557" xml:space="preserve" y="141.7122" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">2. Register resources </text>
+ <text x="150.4286" xml:space="preserve" y="155.681" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">and request AppMaster </text>
+ <text x="199.911" xml:space="preserve" y="169.6497" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">container</text>
+ <path fill="none" d="M421.5 177.0215 L421.5 204.0303" clip-path="url(#clipPath2)"/>
+ <path d="M421.5 212.0303 L426.5 200.0303 L421.5 203.0303 L416.5 200.0303 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <text x="452.5664" xml:space="preserve" y="198.6543" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">3. Allocate AppMaster Container</text>
+ <path fill="none" d="M473.9872 270.5 L606.0008 270.5" clip-path="url(#clipPath2)"/>
+ <path d="M614.0008 270.5 L602.0008 265.5 L605.0008 270.5 L602.0008 275.5 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <text x="481.2285" xml:space="preserve" y="304.6543" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">4. Allocate Worker</text>
+ <path fill="none" d="M210.991 298.1268 L380.3388 385.3096" clip-path="url(#clipPath2)"/>
+ <path d="M387.4516 388.9714 L379.071 379.0333 L379.4497 384.8519 L374.4938 387.9242 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <text x="207.4763" xml:space="preserve" y="355.1486" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">1. Store Uberjar</text>
+ <text x="201.4939" xml:space="preserve" y="369.1174" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">and configuration</text>
+ <path fill="none" d="M421.5 388.4707 L421.5 336.9957" clip-path="url(#clipPath2)" stroke="gray"/>
+ <path fill="gray" d="M421.5 328.9957 L416.5 340.9957 L421.5 337.9957 L426.5 340.9957 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <path fill="none" d="M453.1718 388.9189 L606.9806 303.5393" clip-path="url(#clipPath2)" stroke="gray"/>
+ <path fill="gray" d="M613.9752 299.6566 L601.0566 301.109 L606.1063 304.0247 L605.91 309.8523 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <path fill="none" d="M719.0037 270.5 L783.0085 270.5" clip-path="url(#clipPath2)"/>
+ <path d="M791.0085 270.5 L779.0085 265.5 L782.0085 270.5 L779.0085 275.5 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <path fill="none" d="M473.6061 389.7075 L783.3726 289.8775" clip-path="url(#clipPath2)" stroke="gray"/>
+ <path fill="gray" d="M790.987 287.4236 L778.0318 286.3455 L782.4208 290.1843 L781.0992 295.8634 Z" clip-path="url(#clipPath2)" stroke="none"/>
+ <text x="542.0724" xml:space="preserve" y="392.1359" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">Always Bootstrap containers with</text>
+ <text x="593.1105" xml:space="preserve" y="406.1046" font-weight="bold" clip-path="url(#clipPath2)" stroke="none">Uberjar and config</text>
+ </g>
+ </g>
+</svg>
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/fig/LICENSE.txt
----------------------------------------------------------------------
diff --git a/docs/setup/fig/LICENSE.txt b/docs/setup/fig/LICENSE.txt
new file mode 100644
index 0000000..35b8673
--- /dev/null
+++ b/docs/setup/fig/LICENSE.txt
@@ -0,0 +1,17 @@
+All image files in the folder and its subfolders are
+licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/fig/flink_on_tez_translation.png
----------------------------------------------------------------------
diff --git a/docs/setup/fig/flink_on_tez_translation.png b/docs/setup/fig/flink_on_tez_translation.png
new file mode 100644
index 0000000..88fa4d5
Binary files /dev/null and b/docs/setup/fig/flink_on_tez_translation.png differ
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/fig/flink_tez_vertex.png
----------------------------------------------------------------------
diff --git a/docs/setup/fig/flink_tez_vertex.png b/docs/setup/fig/flink_tez_vertex.png
new file mode 100644
index 0000000..b469862
Binary files /dev/null and b/docs/setup/fig/flink_tez_vertex.png differ
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/flink_on_tez.md
----------------------------------------------------------------------
diff --git a/docs/setup/flink_on_tez.md b/docs/setup/flink_on_tez.md
new file mode 100644
index 0000000..f68c2a1
--- /dev/null
+++ b/docs/setup/flink_on_tez.md
@@ -0,0 +1,290 @@
+---
+title: "Running Flink on YARN leveraging Tez"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+<a href="#top"></a>
+
+You can run Flink using Tez as an execution environment. Flink on Tez
+is currently included in *flink-staging* in alpha. All classes are
+located in the *org.apache.flink.tez* package.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Why Flink on Tez
+
+[Apache Tez](tez.apache.org) is a scalable data processing
+platform. Tez provides an API for specifying a directed acyclic
+graph (DAG), and functionality for placing the DAG vertices in YARN
+containers, as well as data shuffling. In Flink's architecture,
+Tez is at about the same level as Flink's network stack. While Flink's
+network stack focuses heavily on low latency in order to support
+pipelining, data streaming, and iterative algorithms, Tez
+focuses on scalability and elastic resource usage.
+
+Thus, by replacing Flink's network stack with Tez, users can get scalability
+and elastic resource usage in shared clusters while retaining Flink's
+APIs, optimizer, and runtime algorithms (local sorts, hash tables, etc).
+
+Flink programs can run almost unmodified using Tez as an execution
+environment. Tez supports local execution (e.g., for debugging), and
+remote execution on YARN.
+
+
+## Local execution
+
+The `LocalTezEnvironment` can be used run programs using the local
+mode provided by Tez. This example shows how WordCount can be run using the Tez local mode.
+It is identical to a normal Flink WordCount, except that the `LocalTezEnvironment` is used.
+To run in local Tez mode, you can simply run a Flink on Tez program
+from your IDE (e.g., right click and run).
+
+{% highlight java %}
+public class WordCountExample {
+ public static void main(String[] args) throws Exception {
+ final LocalTezEnvironment env = LocalTezEnvironment.create();
+
+ DataSet<String> text = env.fromElements(
+ "Who's there?",
+ "I think I hear them. Stand, ho! Who's there?");
+
+ DataSet<Tuple2<String, Integer>> wordCounts = text
+ .flatMap(new LineSplitter())
+ .groupBy(0)
+ .sum(1);
+
+ wordCounts.print();
+
+ env.execute("Word Count Example");
+ }
+
+ public static class LineSplitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
+ @Override
+ public void flatMap(String line, Collector<Tuple2<String, Integer>> out) {
+ for (String word : line.split(" ")) {
+ out.collect(new Tuple2<String, Integer>(word, 1));
+ }
+ }
+ }
+}
+{% endhighlight %}
+
+## YARN execution
+
+### Setup
+
+- Install Tez on your Hadoop 2 cluster following the instructions from the
+ [Apache Tez website](http://tez.apache.org/install.html). If you are able to run
+ the examples that ship with Tez, then Tez has been successfully installed.
+
+- Currently, you need to build Flink yourself to obtain Flink on Tez
+ (the reason is a Hadoop version compatibility: Tez releases artifacts
+ on Maven central with a Hadoop 2.6.0 dependency). Build Flink
+ using `mvn -DskipTests clean package -Pinclude-tez -Dhadoop.version=X.X.X -Dtez.version=X.X.X`.
+ Make sure that the Hadoop version matches the version that Tez uses.
+ Obtain the jar file contained in the Flink distribution under
+ `flink-staging/flink-tez/target/flink-tez-x.y.z-flink-fat-jar.jar`
+ and upload it to some directory in HDFS. E.g., to upload the file
+ to the directory `/apps`, execute
+ {% highlight bash %}
+ $ hadoop fs -put /path/to/flink-tez-x.y.z-flink-fat-jar.jar /apps
+ {% endhighlight %}
+
+- Edit the tez-site.xml configuration file, adding an entry that points to the
+ location of the file. E.g., assuming that the file is in the directory `/apps/`,
+ add the following entry to tez-site.xml:
+ {% highlight xml %}
+<property>
+ <name>tez.aux.uris</name>
+ <value>${fs.default.name}/apps/flink-tez-x.y.z-flink-fat-jar.jar</value>
+</property>
+ {% endhighlight %}
+
+- At this point, you should be able to run the pre-packaged examples, e.g., run WordCount:
+ {% highlight bash %}
+ $ hadoop jar /path/to/flink-tez-x.y.z-flink-fat-jar.jar wc hdfs:/path/to/text hdfs:/path/to/output
+ {% endhighlight %}
+
+
+### Packaging your program
+
+Application packaging is currently a bit different than in Flink standalone mode.
+ Flink programs that run on Tez need to be packaged in a "fat jar"
+ file that contain the Flink client. This jar can then be executed via the `hadoop jar` command.
+ An easy way to do that is to use the provided `flink-tez-quickstart` maven archetype.
+ Create a new project as
+
+ {% highlight bash %}
+ $ mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.flink \
+ -DarchetypeArtifactId=flink-tez-quickstart \
+ -DarchetypeVersion={{site.version}}
+ {% endhighlight %}
+
+ and specify the group id, artifact id, version, and package of your project. For example,
+ let us assume the following options: `org.myorganization`, `flink-on-tez`, `0.1`, and `org.myorganization`.
+ You should see the following output on your terminal:
+
+ {% highlight bash %}
+ $ mvn archetype:generate -DarchetypeGroupId=org.apache.flink -DarchetypeArtifactId=flink-tez-quickstart
+ [INFO] Scanning for projects...
+ [INFO]
+ [INFO] ------------------------------------------------------------------------
+ [INFO] Building Maven Stub Project (No POM) 1
+ [INFO] ------------------------------------------------------------------------
+ [INFO]
+ [INFO] >>> maven-archetype-plugin:2.2:generate (default-cli) > generate-sources @ standalone-pom >>>
+ [INFO]
+ [INFO] <<< maven-archetype-plugin:2.2:generate (default-cli) < generate-sources @ standalone-pom <<<
+ [INFO]
+ [INFO] --- maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom ---
+ [INFO] Generating project in Interactive mode
+ [INFO] Archetype [org.apache.flink:flink-tez-quickstart:0.9-SNAPSHOT] found in catalog local
+ Define value for property 'groupId': : org.myorganization
+ Define value for property 'artifactId': : flink-on-tez
+ Define value for property 'version': 1.0-SNAPSHOT: : 0.1
+ Define value for property 'package': org.myorganization: :
+ Confirm properties configuration:
+ groupId: org.myorganization
+ artifactId: flink-on-tez
+ version: 0.1
+ package: org.myorganization
+ Y: : Y
+ [INFO] ----------------------------------------------------------------------------
+ [INFO] Using following parameters for creating project from Archetype: flink-tez-quickstart:0.9-SNAPSHOT
+ [INFO] ----------------------------------------------------------------------------
+ [INFO] Parameter: groupId, Value: org.myorganization
+ [INFO] Parameter: artifactId, Value: flink-on-tez
+ [INFO] Parameter: version, Value: 0.1
+ [INFO] Parameter: package, Value: org.myorganization
+ [INFO] Parameter: packageInPathFormat, Value: org/myorganization
+ [INFO] Parameter: package, Value: org.myorganization
+ [INFO] Parameter: version, Value: 0.1
+ [INFO] Parameter: groupId, Value: org.myorganization
+ [INFO] Parameter: artifactId, Value: flink-on-tez
+ [INFO] project created from Archetype in dir: /Users/kostas/Dropbox/flink-tez-quickstart-test/flink-on-tez
+ [INFO] ------------------------------------------------------------------------
+ [INFO] BUILD SUCCESS
+ [INFO] ------------------------------------------------------------------------
+ [INFO] Total time: 44.130 s
+ [INFO] Finished at: 2015-02-26T17:59:45+01:00
+ [INFO] Final Memory: 15M/309M
+ [INFO] ------------------------------------------------------------------------
+ {% endhighlight %}
+
+ The project contains an example called `YarnJob.java` that provides the skeleton
+ for a Flink-on-Tez job. Program execution is currently done using Hadoop's `ProgramDriver`,
+ see the `Driver.java` class for an example. Create the fat jar using
+ `mvn -DskipTests clean package`. The resulting jar will be located in the `target/` directory.
+ You can now execute a job as follows:
+
+ {% highlight bash %}
+$ mvn -DskipTests clean package
+$ hadoop jar flink-on-tez/target/flink-on-tez-0.1-flink-fat-jar.jar yarnjob [command-line parameters]
+ {% endhighlight %}
+
+ Flink programs that run on YARN using Tez as an execution engine need to use the `RemoteTezEnvironment` and
+ register the class that contains the `main` method with that environment:
+ {% highlight java %}
+ public class WordCountExample {
+ public static void main(String[] args) throws Exception {
+ final RemoteTezEnvironment env = RemoteTezEnvironment.create();
+
+ DataSet<String> text = env.fromElements(
+ "Who's there?",
+ "I think I hear them. Stand, ho! Who's there?");
+
+ DataSet<Tuple2<String, Integer>> wordCounts = text
+ .flatMap(new LineSplitter())
+ .groupBy(0)
+ .sum(1);
+
+ wordCounts.print();
+
+ env.registerMainClass(WordCountExample.class);
+ env.execute("Word Count Example");
+ }
+
+ public static class LineSplitter implements FlatMapFunction<String, Tuple2<String, Integer>> {
+ @Override
+ public void flatMap(String line, Collector<Tuple2<String, Integer>> out) {
+ for (String word : line.split(" ")) {
+ out.collect(new Tuple2<String, Integer>(word, 1));
+ }
+ }
+ }
+ }
+ {% endhighlight %}
+
+
+## How it works
+
+Flink on Tez reuses the Flink APIs, the Flink optimizer,
+and the Flink local runtime, including Flink's hash table and sort implementations. Tez
+replaces Flink's network stack and control plan, and is responsible for scheduling and
+network shuffles.
+
+The figure below shows how a Flink program passes through the Flink stack and generates
+a Tez DAG (instead of a JobGraph that would be created using normal Flink execution).
+
+<div style="text-align: center;">
+<img src="fig/flink_on_tez_translation.png" alt="Translation of a Flink program to a Tez DAG." height="600px" vspace="20px" style="text-align: center;"/>
+</div>
+
+All local processing, including memory management, sorting, and hashing is performed by
+Flink as usual. Local processing is encapsulated in Tez vertices, as seen in the figure
+below. Tez vertices are connected by edges. Tez is currently based on a key-value data
+model. In the current implementation, the elements that are processed by Flink operators
+are wrapped inside Tez values, and the Tez key field is used to indicate the index of the target task
+that the elements are destined to.
+
+<div style="text-align: center;">
+<img src="fig/flink_tez_vertex.png" alt="Encapsulation of Flink runtime inside Tez vertices." height="200px" vspace="20px" style="text-align: center;"/>
+</div>
+
+## Limitations
+
+Currently, Flink on Tez does not support all features of the Flink API. We are working
+to enable all of the missing features listed below. In the meantime, if your project depends on these features, we suggest
+to use [Flink on YARN]({{site.baseurl}}/yarn_setup.html) or [Flink standalone]({{site.baseurl}}/setup_quickstart.html).
+
+The following features are currently missing.
+
+- Dedicated client: jobs need to be submitted via Hadoop's command-line client
+
+- Self-joins: currently binary operators that receive the same input are not supported due to
+ [TEZ-1190](https://issues.apache.org/jira/browse/TEZ-1190).
+
+- Iterative programs are currently not supported.
+
+- Broadcast variables are currently not supported.
+
+- Accummulators and counters are currently not supported.
+
+- Performance: The current implementation has not been heavily tested for performance, and misses several optimizations,
+ including task chaining.
+
+- Streaming API: Streaming programs will not currently compile to Tez DAGs.
+
+- Scala API: The current implementation has only been tested with the Java API.
+
+
+
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/gce_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/gce_setup.md b/docs/setup/gce_setup.md
new file mode 100644
index 0000000..c7316ef
--- /dev/null
+++ b/docs/setup/gce_setup.md
@@ -0,0 +1,89 @@
+---
+title: "Google Compute Engine Setup"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+This documentation provides instructions on how to setup Flink fully
+automatically with Hadoop 1 or Hadoop 2 on top of a
+[Google Compute Engine](https://cloud.google.com/compute/) cluster. This is made
+possible by Google's [bdutil](https://cloud.google.com/hadoop/bdutil) which
+starts a cluster and deploys Flink with Hadoop. To get started, just follow the
+steps below.
+
+* This will be replaced by the TOC
+{:toc}
+
+# Prerequisites
+
+## Install Google Cloud SDK
+
+Please follow the instructions on how to setup the
+[Google Cloud SDK](https://cloud.google.com/sdk/).
+
+## Install bdutil
+
+At the moment, there is no bdutil release yet which includes the Flink
+extension. However, you can get the latest version of bdutil with Flink support
+from [GitHub](https://github.com/GoogleCloudPlatform/bdutil):
+
+ git clone https://github.com/GoogleCloudPlatform/bdutil.git
+
+After you have downloaded the source, change into the newly created `bdutil`
+directory and continue with the next steps.
+
+# Deploying Flink on Google Compute Engine
+
+## Set up a bucket
+
+If you have not done so, create a bucket for the bdutil config and
+staging files. A new bucket can be created with gsutil:
+
+ gsutil mb gs://<bucket_name>
+
+
+## Adapt the bdutil config
+
+To deploy Flink with bdutil, adapt at least the following variables in
+bdutil_env.sh.
+
+ CONFIGBUCKET="<bucket_name>"
+ PROJECT="<compute_engine_project_name>"
+ NUM_WORKERS=<number_of_workers>
+
+## Adapt the Flink config
+
+bdutil's Flink extension handles the configuration for you. You may additionally
+adjust configuration variables in `extensions/flink/flink_env.sh`. If you want
+to make further configuration, please take a look at
+[configuring Flink](config.md). You will have to restart Flink after changing
+its configuration using `bin/stop-cluster` and `bin/start-cluster`.
+
+## Bring up a cluster with Flink
+
+To bring up the Flink cluster on Google Compute Engine, execute:
+
+ ./bdutil -e extensions/flink/flink_env.sh deploy
+
+## Run a Flink example job:
+
+ ./bdutil shell
+ cd /home/hadoop/flink-install/bin
+ ./flink run ../examples/flink-java-examples-*-WordCount.jar gs://dataflow-samples/shakespeare/othello.txt gs://<bucket_name>/output
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/index.md
----------------------------------------------------------------------
diff --git a/docs/setup/index.md b/docs/setup/index.md
new file mode 100644
index 0000000..57341ec
--- /dev/null
+++ b/docs/setup/index.md
@@ -0,0 +1,21 @@
+---
+title: "Setup"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/local_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/local_setup.md b/docs/setup/local_setup.md
new file mode 100644
index 0000000..00e9abf
--- /dev/null
+++ b/docs/setup/local_setup.md
@@ -0,0 +1,138 @@
+---
+title: "Local Setup"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This documentation is intended to provide instructions on how to run Flink locally on a single machine.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Download
+
+Go to the [downloads page]({{ site.download_url}}) and get the ready to run package. If you want to interact with Hadoop (e.g. HDFS or HBase), make sure to pick the Flink package **matching your Hadoop version**. When in doubt or you plan to just work with the local file system pick the package for Hadoop 1.2.x.
+
+## Requirements
+
+Flink runs on **Linux**, **Mac OS X** and **Windows**. The only requirement for a local setup is **Java 1.6.x** or higher. The following manual assumes a *UNIX-like environment*, for Windows see [Flink on Windows](#flink-on-windows).
+
+You can check the correct installation of Java by issuing the following command:
+
+~~~bash
+java -version
+~~~
+
+The command should output something comparable to the following:
+
+~~~bash
+java version "1.6.0_22"
+Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
+Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
+~~~
+
+## Configuration
+
+**For local mode Flink is ready to go out of the box and you don't need to change the default configuration.**
+
+The out of the box configuration will use your default Java installation. You can manually set the environment variable `JAVA_HOME` or the configuration key `env.java.home` in `conf/flink-conf.yaml` if you want to manually override the Java runtime to use. Consult the [configuration page](config.html) for further details about configuring Flink.
+
+## Starting Flink
+
+**You are now ready to start Flink.** Unpack the downloaded archive and change to the newly created `flink` directory. There you can start Flink in local mode:
+
+~~~bash
+$ tar xzf flink-*.tgz
+$ cd flink
+$ bin/start-local.sh
+Starting job manager
+~~~
+
+You can check that the system is running by checking the log files in the `logs` directory:
+
+~~~bash
+$ tail log/flink-*-jobmanager-*.log
+INFO ... - Initializing memory manager with 409 megabytes of memory
+INFO ... - Trying to load org.apache.flinknephele.jobmanager.scheduler.local.LocalScheduler as scheduler
+INFO ... - Setting up web info server, using web-root directory ...
+INFO ... - Web info server will display information about nephele job-manager on localhost, port 8081.
+INFO ... - Starting web info server for JobManager on port 8081
+~~~
+
+The JobManager will also start a web frontend on port 8081, which you can check with your browser at `http://localhost:8081`.
+
+## Flink on Windows
+
+If you want to run Flink on Windows you need to download, unpack and configure the Flink archive as mentioned above. After that you can either use the **Windows Batch** file (`.bat`) or use **Cygwin** to run the Flink Jobmanager.
+
+### Starting with Windows Batch Files
+
+To start Flink in local mode from the *Windows Batch*, open the command window, navigate to the `bin/` directory of Flink and run `start-local.bat`.
+
+Note: The ``bin`` folder of your Java Runtime Environment must be included in Window's ``%PATH%`` variable. Follow this [guide](http://www.java.com/en/download/help/path.xml) to add Java to the ``%PATH%`` variable.
+
+~~~bash
+$ cd flink
+$ cd bin
+$ start-local.bat
+Starting Flink job manager. Webinterface by default on http://localhost:8081/.
+Do not close this batch window. Stop job manager by pressing Ctrl+C.
+~~~
+
+After that, you need to open a second terminal to run jobs using `flink.bat`.
+
+### Starting with Cygwin and Unix Scripts
+
+With *Cygwin* you need to start the Cygwin Terminal, navigate to your Flink directory and run the `start-local.sh` script:
+
+~~~bash
+$ cd flink
+$ bin/start-local.sh
+Starting Nephele job manager
+~~~
+
+### Installing Flink from Git
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similiar to this one:
+
+~~~bash
+c:/flink/bin/start-local.sh: line 30: $'\r': command not found
+~~~
+
+This error occurs, because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is, that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
+
+~~~bash
+cd; pwd
+~~~
+
+It will return a path under the Cygwin root path.
+
+2. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you have to create it)
+
+~~~bash
+export SHELLOPTS
+set -o igncr
+~~~
+
+Save the file and open a new bash shell.
+
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup/yarn_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/yarn_setup.md b/docs/setup/yarn_setup.md
new file mode 100644
index 0000000..230b8f9
--- /dev/null
+++ b/docs/setup/yarn_setup.md
@@ -0,0 +1,264 @@
+---
+title: "YARN Setup"
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* This will be replaced by the TOC
+{:toc}
+
+## Quickstart: Start a long-running Flink cluster on YARN
+
+Start a YARN session with 4 Task Managers (each with 4 GB of Heapspace):
+
+~~~bash
+wget {{ site.download_url_hadoop2 }}
+tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
+cd flink-{{ site.version }}/
+./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096
+~~~
+
+Specify the `-s` flag for the number of processing slots per Task Manager. We recommend to set the number of slots to the number of processors per machine.
+
+Once the session has been started, you can submit jobs to the cluster using the `./bin/flink` tool.
+
+## Quickstart: Run a Flink job on YARN
+
+~~~bash
+wget {{ site.download_url_hadoop2 }}
+tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
+cd flink-{{ site.version }}/
+./bin/flink -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-{{ site.version }}-WordCount.jar
+~~~
+
+## Apache Flink on Hadoop YARN using a YARN Session
+
+Apache [Hadoop YARN](http://hadoop.apache.org/) is a cluster resource management framework. It allows to run various distributed applications on top of a cluster. Flink runs on YARN next to other applications. Users do not have to setup or install anything if there is already a YARN setup.
+
+**Requirements**
+
+- Apache Hadoop 2.2
+- HDFS (Hadoop Distributed File System) (or another distributed file system supported by Hadoop)
+
+If you have troubles using the Flink YARN client, have a look in the [FAQ section](faq.html).
+
+### Start Flink Session
+
+Follow these instructions to learn how to launch a Flink Session within your YARN cluster.
+
+A session will start all required Flink services (JobManager and TaskManagers) so that you can submit programs to the cluster. Note that you can run multiple programs per session.
+
+#### Download Flink for YARN
+
+Download the YARN tgz package on the [download page]({{site.baseurl}}/downloads.html). It contains the required files.
+
+Extract the package using:
+
+~~~bash
+tar xvzf flink-{{ site.version }}-bin-hadoop2.tgz
+cd flink-{{site.version }}/
+~~~
+
+If you want to build the YARN .tgz file from sources, follow the [build instructions](building.html). You can find the result of the build in `flink-dist/target/flink-{{ site.version }}-bin/flink-{{ site.version }}/` (*Note: The version might be different for you* ).
+
+
+#### Start a Session
+
+Use the following command to start a session
+
+~~~bash
+./bin/yarn-session.sh
+~~~
+
+This command will show you the following overview:
+
+~~~bash
+Usage:
+ Required
+ -n,--container <arg> Number of YARN container to allocate (=Number of Task Managers)
+ Optional
+ -D <arg> Dynamic properties
+ -d,--detached Start detached
+ -jm,--jobManagerMemory <arg> Memory for JobManager Container [in MB]
+ -q,--query Display available YARN resources (memory, cores)
+ -qu,--queue <arg> Specify YARN queue.
+ -s,--slots <arg> Number of slots per TaskManager
+ -tm,--taskManagerMemory <arg> Memory per TaskManager Container [in MB]
+~~~
+
+Please note that the Client requires the `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment variable to be set to read the YARN and HDFS configuration.
+
+**Example:** Issue the following command to allocate 10 Task Managers, with 8 GB of memory and 32 processing slots each:
+
+~~~bash
+./bin/yarn-session.sh -n 10 -tm 8192 -s 32
+~~~
+
+The system will use the configuration in `conf/flink-config.yaml`. Please follow our [configuration guide](config.html) if you want to change something.
+
+Flink on YARN will overwrite the following configuration parameters `jobmanager.rpc.address` (because the JobManager is always allocated at different machines), `taskmanager.tmp.dirs` (we are using the tmp directories given by YARN) and `parallelism.default` if the number of slots has been specified.
+
+If you don't want to change the configuration file to set configuration parameters, there is the option to pass dynamic properties via the `-D` flag. So you can pass parameters this way: `-Dfs.overwrite-files=true -Dtaskmanager.network.numberOfBuffers=16368`.
+
+The example invocation starts 11 containers, since there is one additional container for the ApplicationMaster and Job Manager.
+
+Once Flink is deployed in your YARN cluster, it will show you the connection details of the Job Manager.
+
+Stop the YARN session by stopping the unix process (using CTRL+C) or by entering 'stop' into the client.
+
+#### Detached YARN session
+
+If you do not want to keep the Flink YARN client running all the time, its also possible to start a *detached* YARN session.
+The parameter for that is called `-d` or `--detached`.
+
+In that case, the Flink YARN client will only submit Flink to the cluster and then close itself.
+Note that in this case its not possible to stop the YARN session using Flink.
+
+Use the YARN utilities (`yarn application -kill <appId`) to stop the YARN session.
+
+
+### Submit Job to Flink
+
+Use the following command to submit a Flink program to the YARN cluster:
+
+~~~bash
+./bin/flink
+~~~
+
+Please refer to the documentation of the [commandline client](cli.html).
+
+The command will show you a help menu like this:
+
+~~~bash
+[...]
+Action "run" compiles and runs a program.
+
+ Syntax: run [OPTIONS] <jar-file> <arguments>
+ "run" action arguments:
+ -c,--class <classname> Class with the program entry point ("main"
+ method or "getPlan()" method. Only needed
+ if the JAR file does not specify the class
+ in its manifest.
+ -m,--jobmanager <host:port> Address of the JobManager (master) to
+ which to connect. Use this flag to connect
+ to a different JobManager than the one
+ specified in the configuration.
+ -p,--parallelism <parallelism> The parallelism with which to run the
+ program. Optional flag to override the
+ default value specified in the
+ configuration
+~~~
+
+Use the *run* action to submit a job to YARN. The client is able to determine the address of the JobManager. In the rare event of a problem, you can also pass the JobManager address using the `-m` argument. The JobManager address is visible in the YARN console.
+
+**Example**
+
+~~~bash
+wget -O apache-license-v2.txt http://www.apache.org/licenses/LICENSE-2.0.txt
+hadoop fs -copyFromLocal LICENSE-2.0.txt hdfs:/// ...
+./bin/flink run ./examples/flink-java-examples-{{site.version }}-WordCount.jar \
+ hdfs:///..../apache-license-v2.txt hdfs:///.../wordcount-result.txt
+~~~
+
+If there is the following error, make sure that all TaskManagers started:
+
+~~~bash
+Exception in thread "main" org.apache.flink.compiler.CompilerException:
+ Available instances could not be determined from job manager: Connection timed out.
+~~~
+
+You can check the number of TaskManagers in the JobManager web interface. The address of this interface is printed in the YARN session console.
+
+If the TaskManagers do not show up after a minute, you should investigate the issue using the log files.
+
+
+## Run a single Flink job on Hadoop YARN
+
+The documentation above describes how to start a Flink cluster within a Hadoop YARN environment.
+It is also possible to launch Flink within YARN only for executing a single job.
+
+Please note that the client then expects the `-yn` value to be set (number of TaskManagers).
+
+***Example:***
+
+~~~bash
+./bin/flink run -m yarn-cluster -yn 2 ./examples/flink-java-examples-{{site.version }}-WordCount.jar
+~~~
+
+The command line options of the YARN session are also available with the `./bin/flink` tool. They are prefixed with a `y` or `yarn` (for the long argument options).
+
+
+## Recovery behavior of Flink on YARN
+
+Flink's YARN client has the following configuration parameters to control how to behave in case of container failures. These parameters can be set either from the `conf/flink-conf.yaml` or when starting the YARN session, using `-D` parameters.
+
+- `yarn.reallocate-failed`: This parameter controls whether Flink should reallocate failed TaskManager containers. Default: true
+- `yarn.maximum-failed-containers`: The maximum number of failed containers the ApplicationMaster accepts until it fails the YARN session. Default: The number of initally requested TaskManagers (`-n`).
+- `yarn.application-attempts`: The number of ApplicationMaster (+ its TaskManager containers) attempts. If this value is set to 1 (default), the entire YARN session will fail when the Application master fails. Higher values specify the number of restarts of the ApplicationMaster by YARN.
+
+
+## Debugging a failed YARN session
+
+There are many reasons why a Flink YARN session deployment can fail. A misconfigured Hadoop setup (HDFS permissions, YARN configuration), version incompatibilities (running Flink with vanilla Hadoop dependencies on Cloudera Hadoop) or other errors.
+
+### Log Files
+
+In cases where the Flink YARN session fails during the deployment itself, users have to rely on the logging capabilities of Hadoop YARN. The most useful feature for that is the [YARN log aggregation](http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/).
+To enable it, users have to set the `yarn.log-aggregation-enable` property to `true` in the `yarn-site.xml` file.
+Once that is enabled, users can use the following command to retrieve all log files of a (failed) YARN session.
+
+~~~
+yarn logs -applicationId <application ID>
+~~~
+
+Note that it takes a few seconds after the session has finished until the logs show up.
+
+### YARN Client console & Webinterfaces
+
+The Flink YARN client also prints error messages in the terminal if errors occur during runtime (for example if a TaskManager stops working after some time).
+
+In addition to that, there is the YARN Resource Manager webinterface (by default on port 8088). The port of the Resource Manager web interface is determined by the `yarn.resourcemanager.webapp.address` configuration value.
+
+It allows to access log files for running YARN applications and shows diagnostics for failed apps.
+
+
+## Build YARN client for a specific Hadoop version
+
+Users using Hadoop distributions from companies like Hortonworks, Cloudera or MapR might have to build Flink against their specific versions of Hadoop (HDFS) and YARN. Please read the [build instructions](building.html) for more details.
+
+
+## Background / Internals
+
+This section briefly describes how Flink and YARN interact.
+
+<img src="fig/FlinkOnYarn.svg" class="img-responsive">
+
+The YARN client needs to access the Hadoop configuration to connect to the YARN resource manager and to HDFS. It determines the Hadoop configuration using the following strategy:
+
+* Test if `YARN_CONF_DIR`, `HADOOP_CONF_DIR` or `HADOOP_CONF_PATH` are set (in that order). If one of these variables are set, they are used to read the configuration.
+* If the above strategy fails (this should not be the case in a correct YARN setup), the client is using the `HADOOP_HOME` environment variable. If it is set, the client tries to access `$HADOOP_HOME/etc/hadoop` (Hadoop 2) and `$HADOOP_HOME/conf` (Hadoop 1).
+
+When starting a new Flink YARN session, the client first checks if the requested resources (containers and memory) are available. After that, it uploads a jar that contains Flink and the configuration to HDFS (step 1).
+
+The next step of the client is to request (step 2) a YARN container to start the *ApplicationMaster* (step 3). Since the client registered the configuration and jar-file as a resource for the container, the NodeManager of YARN running on that particular machine will take care of preparing the container (e.g. downloading the files). Once that has finished, the *ApplicationMaster* (AM) is started.
+
+The *JobManager* and AM are running in the same container. Once they successfully started, the AM knows the address of the JobManager (its own host). It is generating a new Flink configuration file for the TaskManagers (so that they can connect to the JobManager). The file is also uploaded to HDFS. Additionally, the *AM* container is also serving Flink's web interface. The ports Flink is using for its services are the standard ports configured by the user + the application id as an offset. This allows users to execute multiple Flink YARN sessions in parallel.
+
+After that, the AM starts allocating the containers for Flink's TaskManagers, which will download the jar file and the modified configuration from the HDFS. Once these steps are completed, Flink is set up and ready to accept Jobs.
+
http://git-wip-us.apache.org/repos/asf/flink/blob/f1ee90cc/docs/setup_quickstart.md
----------------------------------------------------------------------
diff --git a/docs/setup_quickstart.md b/docs/setup_quickstart.md
deleted file mode 100644
index 5a19d83..0000000
--- a/docs/setup_quickstart.md
+++ /dev/null
@@ -1,155 +0,0 @@
----
-title: "Quickstart: Setup"
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-* This will be replaced by the TOC
-{:toc}
-
-Get Flink up and running in a few simple steps.
-
-## Requirements
-
-Flink runs on __Linux, Mac OS X, and Windows__. To be able to run Flink, the
-only requirement is to have a working __Java 6.x__ (or higher)
-installation. Windows users, please take a look at the
-[Flink on Windows](local_setup.html#flink-on-windows) guide which describes
-how to run Flink on Windows for local setups.
-
-## Download
-Download the ready to run binary package. Choose the Flink distribution that __matches your Hadoop version__. If you are unsure which version to choose or you just want to run locally, pick the package for Hadoop 1.2.
-
-<ul class="nav nav-tabs">
- <li class="active"><a href="#bin-hadoop1" data-toggle="tab">Hadoop 1.2</a></li>
- <li><a href="#bin-hadoop2" data-toggle="tab">Hadoop 2 (YARN)</a></li>
-</ul>
-<p>
-<div class="tab-content text-center">
- <div class="tab-pane active" id="bin-hadoop1">
- <a class="btn btn-info btn-lg" onclick="_gaq.push(['_trackEvent','Action','download-quickstart-setup-1',this.href]);" href="{{site.FLINK_DOWNLOAD_URL_HADOOP1_STABLE}}"><i class="icon-download"> </i> Download Flink for Hadoop 1.2</a>
- </div>
- <div class="tab-pane" id="bin-hadoop2">
- <a class="btn btn-info btn-lg" onclick="_gaq.push(['_trackEvent','Action','download-quickstart-setup-2',this.href]);" href="{{site.FLINK_DOWNLOAD_URL_HADOOP2_STABLE}}"><i class="icon-download"> </i> Download Flink for Hadoop 2</a>
- </div>
-</div>
-</p>
-
-
-## Start
-
-1. Go to the download directory.
-2. Unpack the downloaded archive.
-3. Start Flink.
-
-
-~~~bash
-$ cd ~/Downloads # Go to download directory
-$ tar xzf flink-*.tgz # Unpack the downloaded archive
-$ cd flink-{{site.FLINK_VERSION_SHORT}}
-$ bin/start-local.sh # Start Flink
-~~~
-
-Check the __JobManager's web frontend__ at [http://localhost:8081](http://localhost:8081) and make
-sure everything is up and running.
-
-## Run Example
-
-Run the __Word Count example__ to see Flink at work.
-
-* __Download test data__:
-
- ~~~bash
- $ wget -O hamlet.txt http://www.gutenberg.org/cache/epub/1787/pg1787.txt
- ~~~
-
-* You now have a text file called _hamlet.txt_ in your working directory.
-* __Start the example program__:
-
- ~~~bash
- $ bin/flink run ./examples/flink-java-examples-{{site.FLINK_VERSION_SHORT}}-WordCount.jar file://`pwd`/hamlet.txt file://`pwd`/wordcount-result.txt
- ~~~
-
-* You will find a file called __wordcount-result.txt__ in your current directory.
-
-
-## Cluster Setup
-
-__Running Flink on a cluster__ is as easy as running it locally. Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes lets you use our scripts to control
-everything.
-
-1. Copy the unpacked __flink__ directory from the downloaded archive to the same file system path
-on each node of your setup.
-2. Choose a __master node__ (JobManager) and set the `jobmanager.rpc.address` key in
-`conf/flink-conf.yaml` to its IP or hostname. Make sure that all nodes in your cluster have the same
-`jobmanager.rpc.address` configured.
-3. Add the IPs or hostnames (one per line) of all __worker nodes__ (TaskManager) to the slaves files
-in `conf/slaves`.
-
-You can now __start the cluster__ at your master node with `bin/start-cluster.sh`.
-
-
-The following __example__ illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files, which need to be accessible at the same path on all machines:
-
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="img/quickstart_cluster.png" style="width: 85%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/slaves</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
-
-Have a look at the [Configuration](config.html) section of the documentation to see other available configuration options.
-For Flink to run efficiently, a few configuration values need to be set.
-
-In particular,
-
- * the amount of available memory per TaskManager (`taskmanager.heap.mb`),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`taskmanager.tmp.dirs`)
-
-
-are very important configuration values.
-
-## Flink on YARN
-You can easily deploy Flink on your existing __YARN cluster__.
-
-1. Download the __Flink YARN package__ with the YARN client: [Flink for YARN]({{site.FLINK_DOWNLOAD_URL_YARN_STABLE}})
-2. Make sure your __HADOOP_HOME__ (or _YARN_CONF_DIR_ or _HADOOP_CONF_DIR_) __environment variable__ is set to read your YARN and HDFS configuration.
-3. Run the __YARN client__ with: `./bin/yarn-session.sh`. You can run the client with options `-n 10 -tm 8192` to allocate 10 TaskManagers with 8GB of memory each.
-
-For __more detailed instructions__, check out the programming Guides and examples.
\ No newline at end of file