You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/12/09 09:56:06 UTC
[GitHub] [flink] rmetzger opened a new pull request #14346: [FLINK-20354] Rework standalone docs pages
rmetzger opened a new pull request #14346:
URL: https://github.com/apache/flink/pull/14346
## What is the purpose of the change
Update the Standalone Resource Provider pages.
## Brief change log
- Rewrite the main standalone page, merging information from the local and index page.
- Restructure the Docker standalone page to the new resource provider page structure.
- Adjustments to the Kubernetes standalone page. I decided against applying the new page structure because the existing page is already pretty close to the new structure, and there aren't many topics for the reference.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70b61ad7df7f89069a7f3b0a20445e709ce91ac8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841)
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542372701
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -590,11 +608,11 @@ docker service create \
taskmanager
```
-The *job artifacts* must be available in the *JobManager* container, as outlined [here](#start-a-job-cluster).
+The *job artifacts* must be available in the JobManager container, as outlined [here](#application-mode-on-docker).
See also [how to specify the JobManager arguments](#jobmanager-additional-command-line-arguments) to pass them
to the `flink-jobmanager` container.
The example assumes that you run the swarm locally and expects the *job artifacts* to be in `/host/path/to/job/artifacts`.
-It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
+It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
Review comment:
Sorry for this stuff. I should check the diff myself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542200808
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
Review comment:
but isn't it the purpose of the CLI documentation page to show how to pass job-related parameters?
This guide focuses on educating users on the standalone deployment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542204470
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
Review comment:
I would consider setting up a distributed standalone cluster a fairly advanced topic. Most of our users are probably using K8s or YARN for deployment.
I'm concerned that this will confuse new users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
}, {
"hash" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"triggerType" : "PUSH"
}, {
"hash" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847)
* 395f8fc742495a2a4dccf2597123c3d2b1986e02 UNKNOWN
* 8e4e06a171990c09da0438ed1b2432df5ec59989 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1feff43f3fba39641c0403e7968dfca592b400d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708)
* 2ad8d2e75d7f717ecc0f8ec1f29189306f50a214 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
}, {
"hash" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"triggerType" : "PUSH"
}, {
"hash" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"triggerType" : "PUSH"
}, {
"hash" : "293ea9218bb6340b069cb9e55f55052b8ed4e005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "293ea9218bb6340b069cb9e55f55052b8ed4e005",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847)
* 395f8fc742495a2a4dccf2597123c3d2b1986e02 UNKNOWN
* 8e4e06a171990c09da0438ed1b2432df5ec59989 UNKNOWN
* 293ea9218bb6340b069cb9e55f55052b8ed4e005 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540258892
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
+
+
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes Supported on Docker
Review comment:
I think approach 1 makes more sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-744428922
> Are we planning to update the Chinese versions as well? I didn't look through them, yet.
Once the english version is +1, I'll update the ZH transl
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542232468
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
Review comment:
I think it doesn't hurt, and it's more work removing the references than keeping them, so 🤷♂️
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741664794
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit e3b4f3571079e1a8ce9ce01b103fa44714b8e4a2 (Wed Dec 09 09:58:11 UTC 2020)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ad8d2e75d7f717ecc0f8ec1f29189306f50a214 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
}, {
"hash" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847)
* 395f8fc742495a2a4dccf2597123c3d2b1986e02 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
}, {
"hash" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"triggerType" : "PUSH"
}, {
"hash" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"triggerType" : "PUSH"
}, {
"hash" : "293ea9218bb6340b069cb9e55f55052b8ed4e005",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10864",
"triggerID" : "293ea9218bb6340b069cb9e55f55052b8ed4e005",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 395f8fc742495a2a4dccf2597123c3d2b1986e02 UNKNOWN
* 8e4e06a171990c09da0438ed1b2432df5ec59989 UNKNOWN
* 293ea9218bb6340b069cb9e55f55052b8ed4e005 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10864)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542326276
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -149,16 +143,16 @@ the *JobManager* and *TaskManagers*:
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
[job arguments]
- docker run \
+ $ docker run \
--mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
--mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
-* **or extend the Flink image** by writing a custom `Dockerfile`, build it and use it for starting the *JobManager* and *TaskManagers*:
+* **or extend the Flink image** by writing a custom `Dockerfile`, build it and use it for starting the JobManager and TaskManagers:
- *Dockerfile*:
+ *Dockerfile*:
Review comment:
```suggestion
```
It was already mentioned above that this is a Dockerfile.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -590,11 +608,11 @@ docker service create \
taskmanager
```
-The *job artifacts* must be available in the *JobManager* container, as outlined [here](#start-a-job-cluster).
+The *job artifacts* must be available in the JobManager container, as outlined [here](#application-mode-on-docker).
See also [how to specify the JobManager arguments](#jobmanager-additional-command-line-arguments) to pass them
to the `flink-jobmanager` container.
The example assumes that you run the swarm locally and expects the *job artifacts* to be in `/host/path/to/job/artifacts`.
-It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
+It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
Review comment:
```suggestion
It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,206 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#the-start-and-stop-scripts).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as [Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) or [YARN]({% link deployment/resource-providers/yarn.md %}), you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure your system fulfils the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
-## Flink Setup
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
+# (1) Start Cluster
+$ ./bin/start-cluster.sh
-After downloading the latest release, copy the archive to your master node and extract it:
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+$ ./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
+
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a single TaskManager.
+
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+$ cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+$ ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+$ ./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts. Call them multiple times if you want to stop multiple instances, or use `stop-all`:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+$ ./bin/taskmanager.sh stop
+$ ./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Per-Job Mode
-In particular,
+Per-Job Mode is not supported by the the Standalone Cluster.
Review comment:
```suggestion
Per-Job Mode is not supported by the Standalone Cluster.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,21 +198,53 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
+### Per-Job Cluster Mode
+
+Per-Job Cluster Mode is not supported by Docker.
Review comment:
```suggestion
Per-Job Cluster Mode is not supported by Flink on Docker.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,206 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#the-start-and-stop-scripts).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as [Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) or [YARN]({% link deployment/resource-providers/yarn.md %}), you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure your system fulfils the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
-## Flink Setup
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
+# (1) Start Cluster
+$ ./bin/start-cluster.sh
-After downloading the latest release, copy the archive to your master node and extract it:
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+$ ./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
+
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a single TaskManager.
+
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+$ cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+$ ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+$ ./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts. Call them multiple times if you want to stop multiple instances, or use `stop-all`:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+$ ./bin/taskmanager.sh stop
+$ ./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Per-Job Mode
-In particular,
+Per-Job Mode is not supported by the the Standalone Cluster.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+### Session Mode
-are very important configuration values.
+Local deployment in Session Mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
-{% top %}
+## Standalone Cluster Reference
+
+### Configuration
+
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine. In the default configuration, log files are rotated on each start of a Flink service -- older runs of a service will have a number suffixed to the log file.
-### Starting Flink
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink seems to behave wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level.
-Assuming that you are on the master node and inside the Flink directory:
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
+### Component Management Scripts
+
+#### Starting and Stopping a cluster
+
+`bin/start-cluster.sh` and `bin/stop-cluster.sh` rely on `conf/masters` and `conf/workers` to determine the number of cluster component instances.
+
+If password-less SSH access to the listed machines is configured, and they share the same directory structure, the scripts also support starting and stopping instances remotely.
+
+##### Example 1: Start a cluster with 2 TaskManagers locally
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
-
-{% top %}
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+##### Example 2: Start a distributed cluster JobMangers
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
-#### Adding a JobManager
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+`conf/workers` contents:
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+worker1
+worker2
+worker3
{% endhighlight %}
-#### Adding a TaskManager
+Note that the configuration key [jobmanager.rpc.address]({% link deployment/config.md %}#jobmanager-rpc-address) needs to be set to `master1` for this to work.
+
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
+
+#### Starting and Stopping Flink Components
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running on Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
-## High-Availability with Standalone
+ ```bash
+ cd; pwd
+ ```
+
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following (if the file does not exist you will have to create it):
+
+ ```bash
+ $ export SHELLOPTS
+ $ set -o igncr
+ ```
+
+4. Save the file and open a new bash shell.
+
+### Setting up High-Availability
In order to enable HA for a standalone cluster, you have to use the [ZooKeeper HA services]({% link deployment/ha/zookeeper_ha.md %}).
Additionally, you have to configure your cluster to start multiple JobManagers.
-### Masters File (masters)
-
In order to start an HA-cluster configure the *masters* file in `conf/masters`:
- **masters file**: The *masters file* contains all hosts, on which JobManagers are started, and the ports to which the web user interface binds.
<pre>
-jobManagerAddress1:webUIPort1
+master1:webUIPort1
[...]
-jobManagerAddressX:webUIPortX
+masterX:webUIPortX
</pre>
By default, the job manager will pick a *random port* for inter process communication. You can change this via the [high-availability.jobmanager.port]({% link deployment/config.md %}#high-availability-jobmanager-port) key. This key accepts single ports (e.g. `50010`), ranges (`50000-50025`), or a combination of both (`50010,50011,50020-50025,50050-50075`).
-### Example: Standalone Cluster with 2 JobManagers
+#### Example: Standalone HA Cluster with 2 JobManagers
1. **Configure high availability mode and ZooKeeper quorum** in `conf/flink-conf.yaml`:
Review comment:
I think we could remove the formatting for the items.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,21 +198,53 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
+### Per-Job Cluster Mode
+
+Per-Job Cluster Mode is not supported by Docker.
Review comment:
```suggestion
[Per-Job Mode]({% link deployment/index.md %}#per-job-mode) is not supported by Docker.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,21 +198,53 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
+### Per-Job Cluster Mode
Review comment:
```suggestion
### Per-Job Mode on Docker
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,206 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#the-start-and-stop-scripts).
Review comment:
```suggestion
This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#example-2-start-a-distributed-cluster-jobmangers).
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,206 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#the-start-and-stop-scripts).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as [Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) or [YARN]({% link deployment/resource-providers/yarn.md %}), you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure your system fulfils the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
-## Flink Setup
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
+# (1) Start Cluster
+$ ./bin/start-cluster.sh
-After downloading the latest release, copy the archive to your master node and extract it:
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+$ ./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
+
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a single TaskManager.
+
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+$ cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+$ ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+$ ./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts. Call them multiple times if you want to stop multiple instances, or use `stop-all`:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+$ ./bin/taskmanager.sh stop
+$ ./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Per-Job Mode
-In particular,
+Per-Job Mode is not supported by the the Standalone Cluster.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+### Session Mode
-are very important configuration values.
+Local deployment in Session Mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
-{% top %}
+## Standalone Cluster Reference
+
+### Configuration
+
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine. In the default configuration, log files are rotated on each start of a Flink service -- older runs of a service will have a number suffixed to the log file.
-### Starting Flink
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink seems to behave wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level.
-Assuming that you are on the master node and inside the Flink directory:
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
+### Component Management Scripts
+
+#### Starting and Stopping a cluster
+
+`bin/start-cluster.sh` and `bin/stop-cluster.sh` rely on `conf/masters` and `conf/workers` to determine the number of cluster component instances.
+
+If password-less SSH access to the listed machines is configured, and they share the same directory structure, the scripts also support starting and stopping instances remotely.
+
+##### Example 1: Start a cluster with 2 TaskManagers locally
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
-
-{% top %}
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+##### Example 2: Start a distributed cluster JobMangers
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
-#### Adding a JobManager
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+`conf/workers` contents:
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+worker1
+worker2
+worker3
{% endhighlight %}
-#### Adding a TaskManager
+Note that the configuration key [jobmanager.rpc.address]({% link deployment/config.md %}#jobmanager-rpc-address) needs to be set to `master1` for this to work.
+
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
+
+#### Starting and Stopping Flink Components
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running on Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
-## High-Availability with Standalone
+ ```bash
+ cd; pwd
+ ```
+
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following (if the file does not exist you will have to create it):
+
+ ```bash
+ $ export SHELLOPTS
+ $ set -o igncr
+ ```
+
+4. Save the file and open a new bash shell.
+
+### Setting up High-Availability
In order to enable HA for a standalone cluster, you have to use the [ZooKeeper HA services]({% link deployment/ha/zookeeper_ha.md %}).
Additionally, you have to configure your cluster to start multiple JobManagers.
-### Masters File (masters)
-
In order to start an HA-cluster configure the *masters* file in `conf/masters`:
- **masters file**: The *masters file* contains all hosts, on which JobManagers are started, and the ports to which the web user interface binds.
<pre>
-jobManagerAddress1:webUIPort1
+master1:webUIPort1
[...]
-jobManagerAddressX:webUIPortX
+masterX:webUIPortX
</pre>
By default, the job manager will pick a *random port* for inter process communication. You can change this via the [high-availability.jobmanager.port]({% link deployment/config.md %}#high-availability-jobmanager-port) key. This key accepts single ports (e.g. `50010`), ranges (`50000-50025`), or a combination of both (`50010,50011,50020-50025,50050-50075`).
-### Example: Standalone Cluster with 2 JobManagers
+#### Example: Standalone HA Cluster with 2 JobManagers
Review comment:
Can we switch from `<pre/>` tags to "\`\`\`sh \`\`\`"? This way, we would enable syntax highlighting for these code blocks.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,120 +23,114 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
-
-### Image tags
-
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-For example, you can use the following aliases:
+## Getting Started
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster using Docker containers.
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
+### Introduction
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the Docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
Review comment:
```suggestion
You can use the Docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker Compose.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,120 +23,114 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
-
-### Image tags
-
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-For example, you can use the following aliases:
+## Getting Started
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster using Docker containers.
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
+### Introduction
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the Docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}), are described on separate pages.
-We recommend using [Docker Compose]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-compose) or [Docker Swarm]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
-### Start a Session Cluster
+### Starting a Session Cluster on Docker
-A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
-To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after the cluster has been deployed.
+To deploy a *Flink Session cluster* with Docker, you need to start a JobManager container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
```sh
-FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
-docker network create flink-network
+$ FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+$ docker network create flink-network
```
Then we launch the JobManager:
```sh
-docker run \
+$ docker run \
--rm \
--name=jobmanager \
--network flink-network \
- -p 8081:8081 \
+ --publish 8081:8081 \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} jobmanager
```
-and one or more *TaskManager* containers:
+and one or more TaskManager containers:
```sh
-docker run \
+$ docker run \
--rm \
--name=taskmanager \
--network flink-network \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available):
+
+```sh
+$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with `CTRL-C`) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
+
+This allows you to deploy a standalone cluster (Session or Per-Job Mode) in any containerised environment, for example:
Review comment:
```suggestion
This allows you to deploy a standalone cluster (Session or Application Mode) in any containerised environment, for example:
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -304,29 +349,33 @@ as described in [how to run the Flink image](#how-to-run-flink-image).
# create custom_lib.jar
# create custom_plugin.jar
- echo "
- ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/. # enable an optional library
- ln -fs /mnt/custom_lib.jar /opt/flink/lib/. # enable a custom library
+ $ echo "
+ # enable an optional library
+ ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/
+ # enable a custom library
+ ln -fs /mnt/custom_lib.jar /opt/flink/lib/
mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/. # enable an optional plugin
+ # enable an optional plugin
+ ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/
mkdir -p /opt/flink/plugins/custom_plugin
- ln -fs /mnt/custom_plugin.jar /opt/flink/plugins/custom_plugin/. # enable a custom plugin
+ # enable a custom plugin
+ ln -fs /mnt/custom_plugin.jar /opt/flink/plugins/custom_plugin/
/docker-entrypoint.sh <jobmanager|standalone-job|taskmanager>
" > custom_entry_point_script.sh
- chmod 755 custom_entry_point_script.sh
+ $ chmod 755 custom_entry_point_script.sh
- docker run \
+ $ docker run \
--mount type=bind,src=$(pwd),target=/mnt
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} /mnt/custom_entry_point_script.sh
```
* **extend the Flink image** by writing a custom `Dockerfile` and build a custom image:
- *Dockerfile*:
+ *Dockerfile*:
Review comment:
```suggestion
```
I think that's obsolete again.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -344,209 +393,178 @@ as described in [how to run the Flink image](#how-to-run-flink-image).
ENV VAR_NAME value
```
- **Commands for building**:
+ **Commands for building**:
```sh
- docker build -t custom_flink_image .
+ $ docker build --tag custom_flink_image .
# optional push to your docker image registry if you have it,
# e.g. to distribute the custom image to your cluster
- docker push custom_flink_image
+ $ docker push custom_flink_image
```
-
-### Enabling Python
-
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Application cluster](#app-cluster-yml)
+ * [Session cluster](#session-cluster-yml)
- See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
- for usage in the configuration files.
+ See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
+ for usage in the configuration files.
-* Launch a cluster in the foreground
+* Launch a cluster in the foreground (use `-d` for background)
```sh
- docker-compose up
+ $ docker-compose up
```
-* Launch a cluster in the background
+* Scale the cluster up or down to `N` TaskManagers
```sh
- docker-compose up -d
+ $ docker-compose scale taskmanager=<N>
```
-* Scale the cluster up or down to *N TaskManagers*
+* Access the JobManager container
```sh
- docker-compose scale taskmanager=<N>
- ```
-
-* Access the *JobManager* container
-
- ```sh
- docker exec -it $(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}) /bin/sh
+ $ docker exec -it $(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}) /bin/sh
```
* Kill the cluster
```sh
- docker-compose kill
+ $ docker-compose kill
```
* Access Web UI
- When the cluster is running, you can visit the web UI at [http://localhost:8081](http://localhost:8081).
- You can also use the web UI to submit a job to a *Session cluster*.
+ When the cluster is running, you can visit the web UI at [http://localhost:8081](http://localhost:8081).
+ You can also use the web UI to submit a job to a *Session cluster*.
* To submit a job to a *Session cluster* via the command line, you can either
* use [Flink CLI]({% link deployment/cli.md %}) on the host if it is installed:
```sh
- flink run -d -c ${JOB_CLASS_NAME} /job.jar
+ $ ./bin/flink run --detached --class ${JOB_CLASS_NAME} /job.jar
```
- * or copy the JAR to the *JobManager* container and submit the job using the [CLI]({% link deployment/cli.md %}) from there, for example:
+ * or copy the JAR to the JobManager container and submit the job using the [CLI]({% link deployment/cli.md %}) from there, for example:
```sh
- JOB_CLASS_NAME="com.job.ClassName"
- JM_CONTAINER=$(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}))
- docker cp path/to/jar "${JM_CONTAINER}":/job.jar
- docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
+ $ JOB_CLASS_NAME="com.job.ClassName"
+ $ JM_CONTAINER=$(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}))
+ $ docker cp path/to/jar "${JM_CONTAINER}":/job.jar
+ $ docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
```
-### Session Cluster with Docker Compose
+Here, we provide the **docker-compose.yml:** <a id="app-cluster-yml">file</a> for *Application Cluster*.
-**docker-compose.yml:**
+Note: For the application cluster, the artifacts must be available in the Flink containers, check details [here](#application-mode-on-docker).
Review comment:
```suggestion
Note: For the Application Mode cluster, the artifacts must be available in the Flink containers, check details [here](#application-mode-on-docker).
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,120 +23,114 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
-
-### Image tags
-
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-For example, you can use the following aliases:
+## Getting Started
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster using Docker containers.
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
+### Introduction
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the Docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}), are described on separate pages.
-We recommend using [Docker Compose]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-compose) or [Docker Swarm]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
-### Start a Session Cluster
+### Starting a Session Cluster on Docker
-A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
-To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after the cluster has been deployed.
+To deploy a *Flink Session cluster* with Docker, you need to start a JobManager container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
```sh
-FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
-docker network create flink-network
+$ FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+$ docker network create flink-network
```
Then we launch the JobManager:
```sh
-docker run \
+$ docker run \
--rm \
--name=jobmanager \
--network flink-network \
- -p 8081:8081 \
+ --publish 8081:8081 \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} jobmanager
```
-and one or more *TaskManager* containers:
+and one or more TaskManager containers:
```sh
-docker run \
+$ docker run \
--rm \
--name=taskmanager \
--network flink-network \
--env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available):
+
+```sh
+$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with `CTRL-C`) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
+
+This allows you to deploy a standalone cluster (Session or Per-Job Mode) in any containerised environment, for example:
+* manually in a local Docker setup,
+* [in a (native) Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
Review comment:
```suggestion
* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
```
I'd suggest removing `(native)` here as we're referring to the Standalone Kubernetes deployment. ...just to avoid confusion
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -234,14 +260,14 @@ To provide a custom location for the Flink configuration files, you can
* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
```sh
- docker run \
+ $ docker run \
--mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
```
* or add them to your **custom Flink image**, build and run it:
- *Dockerfile*:
+ *Dockerfile*:
Review comment:
```suggestion
```
I think we can remove `Dockerfile:` here as it's mentioned in the item above that it's a "custom Flink image"
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,206 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#the-start-and-stop-scripts).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as [Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) or [YARN]({% link deployment/resource-providers/yarn.md %}), you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure your system fulfils the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
-## Flink Setup
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
+# (1) Start Cluster
+$ ./bin/start-cluster.sh
-After downloading the latest release, copy the archive to your master node and extract it:
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+$ ./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
+
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a single TaskManager.
+
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+$ cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+$ ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+$ ./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts. Call them multiple times if you want to stop multiple instances, or use `stop-all`:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+$ ./bin/taskmanager.sh stop
+$ ./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Per-Job Mode
-In particular,
+Per-Job Mode is not supported by the the Standalone Cluster.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+### Session Mode
-are very important configuration values.
+Local deployment in Session Mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
-{% top %}
+## Standalone Cluster Reference
+
+### Configuration
+
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine. In the default configuration, log files are rotated on each start of a Flink service -- older runs of a service will have a number suffixed to the log file.
-### Starting Flink
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink seems to behave wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level.
-Assuming that you are on the master node and inside the Flink directory:
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
+### Component Management Scripts
+
+#### Starting and Stopping a cluster
+
+`bin/start-cluster.sh` and `bin/stop-cluster.sh` rely on `conf/masters` and `conf/workers` to determine the number of cluster component instances.
+
+If password-less SSH access to the listed machines is configured, and they share the same directory structure, the scripts also support starting and stopping instances remotely.
+
+##### Example 1: Start a cluster with 2 TaskManagers locally
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
-
-{% top %}
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+##### Example 2: Start a distributed cluster JobMangers
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
-#### Adding a JobManager
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+`conf/workers` contents:
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+worker1
+worker2
+worker3
{% endhighlight %}
-#### Adding a TaskManager
+Note that the configuration key [jobmanager.rpc.address]({% link deployment/config.md %}#jobmanager-rpc-address) needs to be set to `master1` for this to work.
+
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
+
+#### Starting and Stopping Flink Components
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running on Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
-## High-Availability with Standalone
+ ```bash
+ cd; pwd
+ ```
+
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following (if the file does not exist you will have to create it):
+
+ ```bash
+ $ export SHELLOPTS
+ $ set -o igncr
+ ```
+
+4. Save the file and open a new bash shell.
+
+### Setting up High-Availability
In order to enable HA for a standalone cluster, you have to use the [ZooKeeper HA services]({% link deployment/ha/zookeeper_ha.md %}).
Additionally, you have to configure your cluster to start multiple JobManagers.
-### Masters File (masters)
-
In order to start an HA-cluster configure the *masters* file in `conf/masters`:
- **masters file**: The *masters file* contains all hosts, on which JobManagers are started, and the ports to which the web user interface binds.
<pre>
-jobManagerAddress1:webUIPort1
+master1:webUIPort1
[...]
-jobManagerAddressX:webUIPortX
+masterX:webUIPortX
</pre>
By default, the job manager will pick a *random port* for inter process communication. You can change this via the [high-availability.jobmanager.port]({% link deployment/config.md %}#high-availability-jobmanager-port) key. This key accepts single ports (e.g. `50010`), ranges (`50000-50025`), or a combination of both (`50010,50011,50020-50025,50050-50075`).
Review comment:
```suggestion
By default, the JobManager will pick a *random port* for inter process communication. You can change this via the [high-availability.jobmanager.port]({% link deployment/config.md %}#high-availability-jobmanager-port) key. This key accepts single ports (e.g. `50010`), ranges (`50000-50025`), or a combination of both (`50010,50011,50020-50025,50050-50075`).
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -344,209 +393,178 @@ as described in [how to run the Flink image](#how-to-run-flink-image).
ENV VAR_NAME value
```
- **Commands for building**:
+ **Commands for building**:
```sh
- docker build -t custom_flink_image .
+ $ docker build --tag custom_flink_image .
# optional push to your docker image registry if you have it,
# e.g. to distribute the custom image to your cluster
- docker push custom_flink_image
+ $ docker push custom_flink_image
```
-
-### Enabling Python
-
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Application cluster](#app-cluster-yml)
+ * [Session cluster](#session-cluster-yml)
- See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
- for usage in the configuration files.
+ See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
+ for usage in the configuration files.
-* Launch a cluster in the foreground
+* Launch a cluster in the foreground (use `-d` for background)
```sh
- docker-compose up
+ $ docker-compose up
```
-* Launch a cluster in the background
+* Scale the cluster up or down to `N` TaskManagers
```sh
- docker-compose up -d
+ $ docker-compose scale taskmanager=<N>
```
-* Scale the cluster up or down to *N TaskManagers*
+* Access the JobManager container
```sh
- docker-compose scale taskmanager=<N>
- ```
-
-* Access the *JobManager* container
-
- ```sh
- docker exec -it $(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}) /bin/sh
+ $ docker exec -it $(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}) /bin/sh
```
* Kill the cluster
```sh
- docker-compose kill
+ $ docker-compose kill
```
* Access Web UI
- When the cluster is running, you can visit the web UI at [http://localhost:8081](http://localhost:8081).
- You can also use the web UI to submit a job to a *Session cluster*.
+ When the cluster is running, you can visit the web UI at [http://localhost:8081](http://localhost:8081).
+ You can also use the web UI to submit a job to a *Session cluster*.
* To submit a job to a *Session cluster* via the command line, you can either
* use [Flink CLI]({% link deployment/cli.md %}) on the host if it is installed:
```sh
- flink run -d -c ${JOB_CLASS_NAME} /job.jar
+ $ ./bin/flink run --detached --class ${JOB_CLASS_NAME} /job.jar
```
- * or copy the JAR to the *JobManager* container and submit the job using the [CLI]({% link deployment/cli.md %}) from there, for example:
+ * or copy the JAR to the JobManager container and submit the job using the [CLI]({% link deployment/cli.md %}) from there, for example:
```sh
- JOB_CLASS_NAME="com.job.ClassName"
- JM_CONTAINER=$(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}))
- docker cp path/to/jar "${JM_CONTAINER}":/job.jar
- docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
+ $ JOB_CLASS_NAME="com.job.ClassName"
+ $ JM_CONTAINER=$(docker ps --filter name=jobmanager --format={% raw %}{{.ID}}{% endraw %}))
+ $ docker cp path/to/jar "${JM_CONTAINER}":/job.jar
+ $ docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
```
-### Session Cluster with Docker Compose
+Here, we provide the **docker-compose.yml:** <a id="app-cluster-yml">file</a> for *Application Cluster*.
Review comment:
```suggestion
Here, we provide the <a id="app-cluster-yml">docker-compose.yml</a> for *Application Cluster*.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540260143
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
Review comment:
I had exactly the same concern, however I didn't manage to rename it, without the navigation falling apart.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1feff43f3fba39641c0403e7968dfca592b400d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708)
* 2ad8d2e75d7f717ecc0f8ec1f29189306f50a214 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-744447218
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542212468
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -23,45 +23,53 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a *Flink Job* and *Session cluster* on [Kubernetes](https://kubernetes.io).
-
* This will be replaced by the TOC
{:toc}
-{% info %} This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes.
-You can find more information on native Kubernetes deployments [here]({% link deployment/resource-providers/native_kubernetes.md %}).
-## Setup Kubernetes
+## Getting Started
+
+This *Getting Started* guide describes how to deploy a *Session cluster* on [Kubernetes](https://kubernetes.io).
+
+### Introduction
+
+This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes, using Flink's standalone deployment.
+We generally recommend new users to deploy Flink on Kubernetes using [native Kubernetes deployments]({% link deployment/resource-providers/native_kubernetes.md %}).
-Please follow [Kubernetes' setup guide](https://kubernetes.io/docs/setup/) in order to deploy a Kubernetes cluster.
-If you want to run Kubernetes locally, we recommend using [MiniKube](https://kubernetes.io/docs/setup/minikube/).
+### Preparation
+
+This guide expects a Kubernetes environment to be present. You can ensure that your Kubernetes setup is working by running a command like `kubectl get nodes`, which lists all connected Kubelets.
+
+If you want to run Kubernetes locally, we recommend using [MiniKube](https://minikube.sigs.k8s.io/docs/start/).
<div class="alert alert-info" markdown="span">
<strong>Note:</strong> If using MiniKube please make sure to execute `minikube ssh 'sudo ip link set docker0 promisc on'` before deploying a Flink cluster.
Otherwise Flink components are not able to self reference themselves through a Kubernetes service.
</div>
-## Flink Docker image
+### Starting a Kubernetes Cluster (Session Mode)
+
+A *Flink Session cluster* is executed as a long-running Kubernetes Deployment. You can run multiple Flink jobs on a *Session cluster*.
+Each job needs to be submitted to the cluster after the cluster has been deployed.
-Before deploying the Flink Kubernetes components, please read [the Flink Docker image documentation]({% link deployment/resource-providers/standalone/docker.md %}),
-[its tags]({% link deployment/resource-providers/standalone/docker.md %}#image-tags), [how to customize the Flink Docker image]({% link deployment/resource-providers/standalone/docker.md %}#customize-flink-image) and
-[enable plugins]({% link deployment/resource-providers/standalone/docker.md %}#using-plugins) to use the image in the Kubernetes definition files.
+A *Flink Session cluster* deployment in Kubernetes has at least three components:
-## Deploy Flink cluster on Kubernetes
+* a *Deployment* which runs a [JobManager]({% link concepts/glossary.md %}#flink-jobmanager)
+* a *Deployment* for a pool of [TaskManagers]({% link concepts/glossary.md %}#flink-taskmanager)
+* a *Service* exposing the *JobManager's* REST and UI ports
-Using [the common resource definitions](#common-cluster-resource-definitions), launch the common cluster components
-with the `kubectl` command:
+Using the file contents provided in the [the common resource definitions](#common-cluster-resource-definitions), create the following files, and create the respective components with the `kubectl` command:
```sh
+ # Configuration and service definition
kubectl create -f flink-configuration-configmap.yaml
kubectl create -f jobmanager-service.yaml
+ # Create the deployments for the cluster
+ kubectl create -f jobmanager-session-deployment.yaml
+ kubectl create -f taskmanager-session-deployment.yaml
```
-Note that you could define your own customized options of `flink-conf.yaml` within `flink-configuration-configmap.yaml`.
-
-Then launch the specific components depending on whether you want to deploy a [Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster.
-
-You can then access the Flink UI via different ways:
+You can then access the Flink UI and submit jobs via different ways:
Review comment:
I'm always a fan of making the getting started guide shorter and easier ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1feff43f3fba39641c0403e7968dfca592b400d9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540269120
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,191 +198,25 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
-
-When you run the Flink containers, there may be a need to customize them.
-The next chapters describe some how-tos of what you can usually customize.
-
-### Configure options
-
-When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
-
-```sh
-FLINK_PROPERTIES="jobmanager.rpc.address: host
-taskmanager.numberOfTaskSlots: 3
-blob.server.port: 6124
-"
-docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
-```
-
-The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
-
-The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
-the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
-
-### Provide custom configuration
-
-The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
-To provide a custom location for the Flink configuration files, you can
-
-* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
-
- ```sh
- docker run \
- --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
- ```
-
-* or add them to your **custom Flink image**, build and run it:
-
- *Dockerfile*:
-
- ```dockerfile
- FROM flink
- ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
- ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
- ```
-
-<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
-The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
-
-### Using plugins
-
-As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
-copied to the correct location in the Flink installation in the Docker container for them to work.
-
-If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
-The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
-
-```sh
- docker run \
- --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
-```
-
-There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
-
-### Switch memory allocator
-
-Flink introduced `jemalloc` as default memory allocator to resolve memory fragmentation problem (please refer to [FLINK-19125](https://issues.apache.org/jira/browse/FLINK-19125)).
-
-You could switch back to use `glibc` as memory allocator to restore the old behavior or if any unexpected memory consumption or problem observed
-(and please report the issue via JIRA or mailing list if you found any), by passing `disable-jemalloc` parameter:
-
-```sh
- docker run <jobmanager|standalone-job|taskmanager> disable-jemalloc
-```
-
-### Advanced customization
-
-There are several ways in which you can further customize the Flink image:
-
-* install custom software (e.g. python)
-* enable (symlink) optional libraries or plugins from `/opt/flink/opt` into `/opt/flink/lib` or `/opt/flink/plugins`
-* add other libraries to `/opt/flink/lib` (e.g. Hadoop)
-* add other plugins to `/opt/flink/plugins`
-
-See also: [How to provide dependencies in the classpath]({% link index.md %}#how-to-provide-dependencies-in-the-classpath).
-
-You can customize the Flink image in several ways:
-
-* **override the container entry point** with a custom script where you can run any bootstrap actions.
-At the end you can call the standard `/docker-entrypoint.sh` script of the Flink image with the same arguments
-as described in [how to run the Flink image](#how-to-run-flink-image).
-
- The following example creates a custom entry point script which enables more libraries and plugins.
- The custom script, custom library and plugin are provided from a mounted volume.
- Then it runs the standard entry point script of the Flink image:
-
- ```sh
- # create custom_lib.jar
- # create custom_plugin.jar
-
- echo "
- ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/. # enable an optional library
- ln -fs /mnt/custom_lib.jar /opt/flink/lib/. # enable a custom library
-
- mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/. # enable an optional plugin
-
- mkdir -p /opt/flink/plugins/custom_plugin
- ln -fs /mnt/custom_plugin.jar /opt/flink/plugins/custom_plugin/. # enable a custom plugin
-
- /docker-entrypoint.sh <jobmanager|standalone-job|taskmanager>
- " > custom_entry_point_script.sh
-
- chmod 755 custom_entry_point_script.sh
-
- docker run \
- --mount type=bind,src=$(pwd),target=/mnt
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} /mnt/custom_entry_point_script.sh
- ```
-
-* **extend the Flink image** by writing a custom `Dockerfile` and build a custom image:
- *Dockerfile*:
+### Session Mode on Docker
- ```dockerfile
- FROM flink
+Local deployment in the session mode has already been described in the [introduction](#starting-a-session-cluster-on-docker) above.
- RUN set -ex; apt-get update; apt-get -y install python
- ADD /host/path/to/flink-conf.yaml /container/local/path/to/custom/conf/flink-conf.yaml
- ADD /host/path/to/log4j.properties /container/local/path/to/custom/conf/log4j.properties
-
- RUN ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/.
-
- RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- RUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
-
- ENV VAR_NAME value
- ```
-
- **Commands for building**:
-
- ```sh
- docker build -t custom_flink_image .
- # optional push to your docker image registry if you have it,
- # e.g. to distribute the custom image to your cluster
- docker push custom_flink_image
- ```
-
-### Enabling Python
-
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Session cluster](#session-cluster-with-docker-compose)
+ * [Application cluster](#application-cluster-with-docker-compose)
- See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
- for usage in the configuration files.
+ See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
+ for usage in the configuration files.
* Launch a cluster in the foreground
Review comment:
I will shorten it a tiny bit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
}, {
"hash" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "395f8fc742495a2a4dccf2597123c3d2b1986e02",
"triggerType" : "PUSH"
}, {
"hash" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e4e06a171990c09da0438ed1b2432df5ec59989",
"triggerType" : "PUSH"
}, {
"hash" : "293ea9218bb6340b069cb9e55f55052b8ed4e005",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10864",
"triggerID" : "293ea9218bb6340b069cb9e55f55052b8ed4e005",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847)
* 395f8fc742495a2a4dccf2597123c3d2b1986e02 UNKNOWN
* 8e4e06a171990c09da0438ed1b2432df5ec59989 UNKNOWN
* 293ea9218bb6340b069cb9e55f55052b8ed4e005 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10864)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542317616
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
Review comment:
This also includes updating the headline as suggested above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542234170
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,12 +198,44 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
+### Per-Job Cluster Mode
+
+Per-Job Cluster Mode is not supported by Docker.
+
+### Session Mode on Docker
+
+Local deployment in the session mode has already been described in the [introduction](#starting-a-session-cluster-on-docker) above.
+
+
+{% top %}
+
+## Flink on Docker Reference
+
+### Image tags
+
+The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
+The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+
+Images for each supported combination of Flink and Scala versions are available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
+
+For example, you can use the following aliases:
+
+* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-When you run the Flink containers, there may be a need to customize them.
-The next chapters describe some how-tos of what you can usually customize.
+<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
+versions (for example `flink:1.11-scala_2.12`).
+This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
+from the versions provided by the docker image.
+
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
-### Configure options
+
+### Passing configuration via environment variables
When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
Review comment:
I'm not sure if that really helps avoiding confusion. I would prefer to keep it this way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540938493
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -590,11 +416,203 @@ docker service create \
taskmanager
```
-The *job artifacts* must be available in the *JobManager* container, as outlined [here](#start-a-job-cluster).
+The *job artifacts* must be available in the *JobManager* container, as outlined [here](#application-mode-on-docker).
See also [how to specify the JobManager arguments](#jobmanager-additional-command-line-arguments) to pass them
to the `flink-jobmanager` container.
The example assumes that you run the swarm locally and expects the *job artifacts* to be in `/host/path/to/job/artifacts`.
It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
{% top %}
+
+## Flink on Docker Reference
+
+### Image tags
+
+The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
+The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+
+Images for each supported combination of Flink and Scala versions are available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
+
+For example, you can use the following aliases:
+
+* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+
+<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
+versions (for example `flink:1.11-scala_2.12`).
+This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
+from the versions provided by the docker image.
+
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+
+### Passing configuration via environment variables
+
+When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
+
+```sh
+FLINK_PROPERTIES="jobmanager.rpc.address: host
+taskmanager.numberOfTaskSlots: 3
+blob.server.port: 6124
+"
+docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
+
+The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
+the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
+
+### Provide custom configuration
+
+The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
+To provide a custom location for the Flink configuration files, you can
+
+* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
+
+ ```sh
+ docker run \
+ --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+ ```
+
+* or add them to your **custom Flink image**, build and run it:
+
+ *Dockerfile*:
+
+ ```dockerfile
+ FROM flink
+ ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
+ ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
+ ```
+
+<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
+The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
+
+### Using filesystem plugins
+
+As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation in the Docker container for them to work.
+
+If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
+The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
+
+```sh
+ docker run \
+ --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
+
+### Enabling Python
Review comment:
Yup, here's the [PR #14367 against Flink's master](https://github.com/apache/flink/pull/14367).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540259319
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -442,7 +270,7 @@ The next chapters show examples of configuration files to run Flink.
docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
```
-### Session Cluster with Docker Compose
+#### Session Cluster with Docker Compose
Review comment:
I'll try it out!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1feff43f3fba39641c0403e7968dfca592b400d9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540870878
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -590,11 +416,203 @@ docker service create \
taskmanager
```
-The *job artifacts* must be available in the *JobManager* container, as outlined [here](#start-a-job-cluster).
+The *job artifacts* must be available in the *JobManager* container, as outlined [here](#application-mode-on-docker).
See also [how to specify the JobManager arguments](#jobmanager-additional-command-line-arguments) to pass them
to the `flink-jobmanager` container.
The example assumes that you run the swarm locally and expects the *job artifacts* to be in `/host/path/to/job/artifacts`.
It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
{% top %}
+
+## Flink on Docker Reference
+
+### Image tags
+
+The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
+The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+
+Images for each supported combination of Flink and Scala versions are available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
+
+For example, you can use the following aliases:
+
+* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+
+<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
+versions (for example `flink:1.11-scala_2.12`).
+This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
+from the versions provided by the docker image.
+
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+
+### Passing configuration via environment variables
+
+When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
+
+```sh
+FLINK_PROPERTIES="jobmanager.rpc.address: host
+taskmanager.numberOfTaskSlots: 3
+blob.server.port: 6124
+"
+docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
+
+The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
+the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
+
+### Provide custom configuration
+
+The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
+To provide a custom location for the Flink configuration files, you can
+
+* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
+
+ ```sh
+ docker run \
+ --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+ ```
+
+* or add them to your **custom Flink image**, build and run it:
+
+ *Dockerfile*:
+
+ ```dockerfile
+ FROM flink
+ ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
+ ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
+ ```
+
+<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
+The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
+
+### Using filesystem plugins
+
+As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation in the Docker container for them to work.
+
+If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
+The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
+
+```sh
+ docker run \
+ --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
+
+### Enabling Python
Review comment:
Will this PR also get opened against Flink?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r539307215
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,191 +198,25 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
-
-When you run the Flink containers, there may be a need to customize them.
-The next chapters describe some how-tos of what you can usually customize.
-
-### Configure options
-
-When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
-
-```sh
-FLINK_PROPERTIES="jobmanager.rpc.address: host
-taskmanager.numberOfTaskSlots: 3
-blob.server.port: 6124
-"
-docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
-```
-
-The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
-
-The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
-the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
-
-### Provide custom configuration
-
-The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
-To provide a custom location for the Flink configuration files, you can
-
-* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
-
- ```sh
- docker run \
- --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
- ```
-
-* or add them to your **custom Flink image**, build and run it:
-
- *Dockerfile*:
-
- ```dockerfile
- FROM flink
- ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
- ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
- ```
-
-<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
-The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
-
-### Using plugins
-
-As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
-copied to the correct location in the Flink installation in the Docker container for them to work.
-
-If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
-The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
-
-```sh
- docker run \
- --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
-```
-
-There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
-
-### Switch memory allocator
-
-Flink introduced `jemalloc` as default memory allocator to resolve memory fragmentation problem (please refer to [FLINK-19125](https://issues.apache.org/jira/browse/FLINK-19125)).
-
-You could switch back to use `glibc` as memory allocator to restore the old behavior or if any unexpected memory consumption or problem observed
-(and please report the issue via JIRA or mailing list if you found any), by passing `disable-jemalloc` parameter:
-
-```sh
- docker run <jobmanager|standalone-job|taskmanager> disable-jemalloc
-```
-
-### Advanced customization
-
-There are several ways in which you can further customize the Flink image:
-
-* install custom software (e.g. python)
-* enable (symlink) optional libraries or plugins from `/opt/flink/opt` into `/opt/flink/lib` or `/opt/flink/plugins`
-* add other libraries to `/opt/flink/lib` (e.g. Hadoop)
-* add other plugins to `/opt/flink/plugins`
-
-See also: [How to provide dependencies in the classpath]({% link index.md %}#how-to-provide-dependencies-in-the-classpath).
-
-You can customize the Flink image in several ways:
-
-* **override the container entry point** with a custom script where you can run any bootstrap actions.
-At the end you can call the standard `/docker-entrypoint.sh` script of the Flink image with the same arguments
-as described in [how to run the Flink image](#how-to-run-flink-image).
-
- The following example creates a custom entry point script which enables more libraries and plugins.
- The custom script, custom library and plugin are provided from a mounted volume.
- Then it runs the standard entry point script of the Flink image:
-
- ```sh
- # create custom_lib.jar
- # create custom_plugin.jar
-
- echo "
- ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/. # enable an optional library
- ln -fs /mnt/custom_lib.jar /opt/flink/lib/. # enable a custom library
-
- mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/. # enable an optional plugin
-
- mkdir -p /opt/flink/plugins/custom_plugin
- ln -fs /mnt/custom_plugin.jar /opt/flink/plugins/custom_plugin/. # enable a custom plugin
-
- /docker-entrypoint.sh <jobmanager|standalone-job|taskmanager>
- " > custom_entry_point_script.sh
-
- chmod 755 custom_entry_point_script.sh
-
- docker run \
- --mount type=bind,src=$(pwd),target=/mnt
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} /mnt/custom_entry_point_script.sh
- ```
-
-* **extend the Flink image** by writing a custom `Dockerfile` and build a custom image:
- *Dockerfile*:
+### Session Mode on Docker
- ```dockerfile
- FROM flink
+Local deployment in the session mode has already been described in the [introduction](#starting-a-session-cluster-on-docker) above.
- RUN set -ex; apt-get update; apt-get -y install python
- ADD /host/path/to/flink-conf.yaml /container/local/path/to/custom/conf/flink-conf.yaml
- ADD /host/path/to/log4j.properties /container/local/path/to/custom/conf/log4j.properties
-
- RUN ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/.
-
- RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- RUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
-
- ENV VAR_NAME value
- ```
-
- **Commands for building**:
-
- ```sh
- docker build -t custom_flink_image .
- # optional push to your docker image registry if you have it,
- # e.g. to distribute the custom image to your cluster
- docker push custom_flink_image
- ```
-
-### Enabling Python
-
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
Review comment:
I would suggest moving the Docker Compose and Docker Swarm section into references section considering that the page is actually focusing on Docker.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
+
+
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes Supported on Docker
Review comment:
I also added unsupported deployment modes on the Mesos page without noticing that the section's name is "Deployment Modes Supported on ...". We went along with it in k8s because we thought it would make it more explicit having a section for each of the three deployment modes and explicitly mentioning that that specific mode is not supported. Considering that we have two options now:
1. Do the same here and rename the headline from "Deployment Modes Supported on Docker" to "Deployment Modes". This would mean that we still have to fix the headline on the Mesos and k8s page accordingly.
2. We stick to the approach you used on this page. That would mean that we would have to remove the respective sections from the k8s and Mesos docs.
Which one do you prefer?
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,191 +198,25 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
-
-When you run the Flink containers, there may be a need to customize them.
-The next chapters describe some how-tos of what you can usually customize.
-
-### Configure options
-
-When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
-
-```sh
-FLINK_PROPERTIES="jobmanager.rpc.address: host
-taskmanager.numberOfTaskSlots: 3
-blob.server.port: 6124
-"
-docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
-```
-
-The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
-
-The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
-the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
-
-### Provide custom configuration
-
-The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
-To provide a custom location for the Flink configuration files, you can
-
-* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
-
- ```sh
- docker run \
- --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
- ```
-
-* or add them to your **custom Flink image**, build and run it:
-
- *Dockerfile*:
-
- ```dockerfile
- FROM flink
- ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
- ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
- ```
-
-<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
-The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
-
-### Using plugins
-
-As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
-copied to the correct location in the Flink installation in the Docker container for them to work.
-
-If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
-The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
-
-```sh
- docker run \
- --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
-```
-
-There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
-
-### Switch memory allocator
-
-Flink introduced `jemalloc` as default memory allocator to resolve memory fragmentation problem (please refer to [FLINK-19125](https://issues.apache.org/jira/browse/FLINK-19125)).
-
-You could switch back to use `glibc` as memory allocator to restore the old behavior or if any unexpected memory consumption or problem observed
-(and please report the issue via JIRA or mailing list if you found any), by passing `disable-jemalloc` parameter:
-
-```sh
- docker run <jobmanager|standalone-job|taskmanager> disable-jemalloc
-```
-
-### Advanced customization
-
-There are several ways in which you can further customize the Flink image:
-
-* install custom software (e.g. python)
-* enable (symlink) optional libraries or plugins from `/opt/flink/opt` into `/opt/flink/lib` or `/opt/flink/plugins`
-* add other libraries to `/opt/flink/lib` (e.g. Hadoop)
-* add other plugins to `/opt/flink/plugins`
-
-See also: [How to provide dependencies in the classpath]({% link index.md %}#how-to-provide-dependencies-in-the-classpath).
-
-You can customize the Flink image in several ways:
-
-* **override the container entry point** with a custom script where you can run any bootstrap actions.
-At the end you can call the standard `/docker-entrypoint.sh` script of the Flink image with the same arguments
-as described in [how to run the Flink image](#how-to-run-flink-image).
-
- The following example creates a custom entry point script which enables more libraries and plugins.
- The custom script, custom library and plugin are provided from a mounted volume.
- Then it runs the standard entry point script of the Flink image:
-
- ```sh
- # create custom_lib.jar
- # create custom_plugin.jar
-
- echo "
- ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/. # enable an optional library
- ln -fs /mnt/custom_lib.jar /opt/flink/lib/. # enable a custom library
-
- mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/. # enable an optional plugin
-
- mkdir -p /opt/flink/plugins/custom_plugin
- ln -fs /mnt/custom_plugin.jar /opt/flink/plugins/custom_plugin/. # enable a custom plugin
-
- /docker-entrypoint.sh <jobmanager|standalone-job|taskmanager>
- " > custom_entry_point_script.sh
-
- chmod 755 custom_entry_point_script.sh
-
- docker run \
- --mount type=bind,src=$(pwd),target=/mnt
- flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} /mnt/custom_entry_point_script.sh
- ```
-
-* **extend the Flink image** by writing a custom `Dockerfile` and build a custom image:
- *Dockerfile*:
+### Session Mode on Docker
- ```dockerfile
- FROM flink
+Local deployment in the session mode has already been described in the [introduction](#starting-a-session-cluster-on-docker) above.
- RUN set -ex; apt-get update; apt-get -y install python
- ADD /host/path/to/flink-conf.yaml /container/local/path/to/custom/conf/flink-conf.yaml
- ADD /host/path/to/log4j.properties /container/local/path/to/custom/conf/log4j.properties
-
- RUN ln -fs /opt/flink/opt/flink-queryable-state-runtime-*.jar /opt/flink/lib/.
-
- RUN mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
- RUN ln -fs /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/plugins/flink-s3-fs-hadoop/.
-
- ENV VAR_NAME value
- ```
-
- **Commands for building**:
-
- ```sh
- docker build -t custom_flink_image .
- # optional push to your docker image registry if you have it,
- # e.g. to distribute the custom image to your cluster
- docker push custom_flink_image
- ```
-
-### Enabling Python
-
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Session cluster](#session-cluster-with-docker-compose)
+ * [Application cluster](#application-cluster-with-docker-compose)
- See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
- for usage in the configuration files.
+ See also [the Flink Docker image tags](#image-tags) and [how to customize the Flink Docker image](#advanced-customization)
+ for usage in the configuration files.
* Launch a cluster in the foreground
Review comment:
I'm not sure whether we have to have this docker-compose tutorial here. I'd say that the `docker-compose` actions can be found in the original `docker-compose` docs. Maybe, we can link them instead.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -442,7 +270,7 @@ The next chapters show examples of configuration files to run Flink.
docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
```
-### Session Cluster with Docker Compose
+#### Session Cluster with Docker Compose
Review comment:
Can't we just have a code block for each of the yaml configurations? Not sure whether the section are necessary here.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -590,11 +416,203 @@ docker service create \
taskmanager
```
-The *job artifacts* must be available in the *JobManager* container, as outlined [here](#start-a-job-cluster).
+The *job artifacts* must be available in the *JobManager* container, as outlined [here](#application-mode-on-docker).
See also [how to specify the JobManager arguments](#jobmanager-additional-command-line-arguments) to pass them
to the `flink-jobmanager` container.
The example assumes that you run the swarm locally and expects the *job artifacts* to be in `/host/path/to/job/artifacts`.
It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
{% top %}
+
+## Flink on Docker Reference
+
+### Image tags
+
+The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
+The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+
+Images for each supported combination of Flink and Scala versions are available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
+
+For example, you can use the following aliases:
+
+* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+
+<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
+versions (for example `flink:1.11-scala_2.12`).
+This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
+from the versions provided by the docker image.
+
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+
+### Passing configuration via environment variables
+
+When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
+
+```sh
+FLINK_PROPERTIES="jobmanager.rpc.address: host
+taskmanager.numberOfTaskSlots: 3
+blob.server.port: 6124
+"
+docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
+
+The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
+the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
+
+### Provide custom configuration
+
+The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
+To provide a custom location for the Flink configuration files, you can
+
+* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
+
+ ```sh
+ docker run \
+ --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+ ```
+
+* or add them to your **custom Flink image**, build and run it:
+
+ *Dockerfile*:
+
+ ```dockerfile
+ FROM flink
+ ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
+ ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
+ ```
+
+<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
+The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
+
+### Using filesystem plugins
+
+As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation in the Docker container for them to work.
+
+If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
+The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
+
+```sh
+ docker run \
+ --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
+
+### Enabling Python
Review comment:
Should we point to the Python CLI that is going to get introduced by @shuiqiangchen in [this PR](https://github.com/XComp/flink/pull/1)?
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -23,7 +23,7 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a *Flink Job* and *Session cluster* on [Kubernetes](https://kubernetes.io).
+This page describes how to deploy a *Flink application cluster* and *Session cluster* on [Kubernetes](https://kubernetes.io).
Review comment:
Couldn't we refactor the documentation here as well sticking to the sections we used for all the other resource provider pages? I have the feeling that it would work here as well 🤔
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
Review comment:
I don't like the "Overview" entry in the menu panel on the left as we're not giving an overview but introduce the basic (or native) deployment of a Standalone cluster. What is your opinion on the naming?
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -442,7 +270,7 @@ The next chapters show examples of configuration files to run Flink.
docker exec -t -i "${JM_CONTAINER}" flink run -d -c ${JOB_CLASS_NAME} /job.jar
```
-### Session Cluster with Docker Compose
+#### Session Cluster with Docker Compose
Review comment:
FYI: What about using collapsible blocks to hide long configuration files:
```
<p>
<a class="btn" data-toggle="collapse" href="#session-cluster-yml" role="button" aria-expanded="false" aria-controls="session-cluster-yml">
docker-compose.yml
</a>
</p>
<div class="collapse" id="session-cluster-yml">
{% highlight yaml %}
version: "2.2"
services:
jobmanager:
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
ports:
- "8081:8081"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
depends_on:
- jobmanager
command: taskmanager
scale: 1
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 2
{% endhighlight %}
</div>
```
I got inspired by [this blog post](https://www.iditect.com/how-to/52576626.html).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542237513
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -268,6 +294,27 @@ The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names sep
Review comment:
I think the text states already "jar file names", and the example is also without a path. In my opinion, this is sufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ad8d2e75d7f717ecc0f8ec1f29189306f50a214 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811)
* 70b61ad7df7f89069a7f3b0a20445e709ce91ac8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
shuiqiangchen commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r539957153
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -590,11 +416,203 @@ docker service create \
taskmanager
```
-The *job artifacts* must be available in the *JobManager* container, as outlined [here](#start-a-job-cluster).
+The *job artifacts* must be available in the *JobManager* container, as outlined [here](#application-mode-on-docker).
See also [how to specify the JobManager arguments](#jobmanager-additional-command-line-arguments) to pass them
to the `flink-jobmanager` container.
The example assumes that you run the swarm locally and expects the *job artifacts* to be in `/host/path/to/job/artifacts`.
It also mounts the host path with the artifacts as a volume to the container's path `/opt/flink/usrlib`.
{% top %}
+
+## Flink on Docker Reference
+
+### Image tags
+
+The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
+The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+
+Images for each supported combination of Flink and Scala versions are available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
+
+For example, you can use the following aliases:
+
+* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
+
+<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
+versions (for example `flink:1.11-scala_2.12`).
+This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
+from the versions provided by the docker image.
+
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
+
+
+### Passing configuration via environment variables
+
+When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
+
+```sh
+FLINK_PROPERTIES="jobmanager.rpc.address: host
+taskmanager.numberOfTaskSlots: 3
+blob.server.port: 6124
+"
+docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+The [`jobmanager.rpc.address`]({% link deployment/config.md %}#jobmanager-rpc-address) option must be configured, others are optional to set.
+
+The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
+the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
+
+### Provide custom configuration
+
+The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image.
+To provide a custom location for the Flink configuration files, you can
+
+* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image:
+
+ ```sh
+ docker run \
+ --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+ ```
+
+* or add them to your **custom Flink image**, build and run it:
+
+ *Dockerfile*:
+
+ ```dockerfile
+ FROM flink
+ ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml
+ ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties
+ ```
+
+<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
+The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
+
+### Using filesystem plugins
+
+As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation in the Docker container for them to work.
+
+If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
+The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
+
+```sh
+ docker run \
+ --env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
+```
+
+There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
+
+### Enabling Python
Review comment:
Maybe we should add a link to guide users to this part for kubernetes application submission which requires users to provide an image in the Python CLI section.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542206249
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+cd; pwd
{% endhighlight %}
-#### Adding a TaskManager
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+export SHELLOPTS
+set -o igncr
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+Save the file and open a new bash shell.
Review comment:
I don't have Windows available either. I'll add the 4. but leave the sourcing out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542142908
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
Review comment:
I think you are right. Stuff runs on windows 😕
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542248744
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -344,64 +389,36 @@ as described in [how to run the Flink image](#how-to-run-flink-image).
ENV VAR_NAME value
```
- **Commands for building**:
+ **Commands for building**:
```sh
docker build -t custom_flink_image .
# optional push to your docker image registry if you have it,
# e.g. to distribute the custom image to your cluster
docker push custom_flink_image
```
-
-### Enabling Python
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Application cluster](#app-cluster-yml)
+ * [Session cluster](#session-cluster-yml)
Review comment:
Okay, I'll revert the collapse-thing.
I don't think adding another indirection to a appending makes sense here, since this is already the bottom of the page.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542203091
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
Review comment:
Good idea! Applied.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1feff43f3fba39641c0403e7968dfca592b400d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70b61ad7df7f89069a7f3b0a20445e709ce91ac8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r542346587
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
Review comment:
Damn, forgot that one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-745080117
Thanks a lot for the extensive review. Merging this now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
XComp commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540976730
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
Review comment:
```suggestion
The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as [Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) or [YARN]({% link deployment/resource-providers/yarn.md %}), you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
Review comment:
```suggestion
Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure your system fulfils the following requirements.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
Review comment:
I would add some remark that `./bin/taskmanager.sh stop` needs to be called as often as the amount of TaskManagers that were started. Each call stops one TaskManager only. Alternatively, we can use `stop-all` instead.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
Review comment:
```suggestion
By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink seems to behave wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
Review comment:
```suggestion
The `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
```
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -23,45 +23,53 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a *Flink Job* and *Session cluster* on [Kubernetes](https://kubernetes.io).
-
* This will be replaced by the TOC
{:toc}
-{% info %} This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes.
-You can find more information on native Kubernetes deployments [here]({% link deployment/resource-providers/native_kubernetes.md %}).
-## Setup Kubernetes
+## Getting Started
+
+This *Getting Started* guide describes how to deploy a *Session cluster* on [Kubernetes](https://kubernetes.io).
+
+### Introduction
+
+This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes, using Flink's standalone deployment.
+We generally recommend new users to deploy Flink on Kubernetes using [native Kubernetes deployments]({% link deployment/resource-providers/native_kubernetes.md %}).
-Please follow [Kubernetes' setup guide](https://kubernetes.io/docs/setup/) in order to deploy a Kubernetes cluster.
-If you want to run Kubernetes locally, we recommend using [MiniKube](https://kubernetes.io/docs/setup/minikube/).
+### Preparation
+
+This guide expects a Kubernetes environment to be present. You can ensure that your Kubernetes setup is working by running a command like `kubectl get nodes`, which lists all connected Kubelets.
+
+If you want to run Kubernetes locally, we recommend using [MiniKube](https://minikube.sigs.k8s.io/docs/start/).
<div class="alert alert-info" markdown="span">
<strong>Note:</strong> If using MiniKube please make sure to execute `minikube ssh 'sudo ip link set docker0 promisc on'` before deploying a Flink cluster.
Otherwise Flink components are not able to self reference themselves through a Kubernetes service.
</div>
-## Flink Docker image
+### Starting a Kubernetes Cluster (Session Mode)
+
+A *Flink Session cluster* is executed as a long-running Kubernetes Deployment. You can run multiple Flink jobs on a *Session cluster*.
+Each job needs to be submitted to the cluster after the cluster has been deployed.
-Before deploying the Flink Kubernetes components, please read [the Flink Docker image documentation]({% link deployment/resource-providers/standalone/docker.md %}),
-[its tags]({% link deployment/resource-providers/standalone/docker.md %}#image-tags), [how to customize the Flink Docker image]({% link deployment/resource-providers/standalone/docker.md %}#customize-flink-image) and
-[enable plugins]({% link deployment/resource-providers/standalone/docker.md %}#using-plugins) to use the image in the Kubernetes definition files.
+A *Flink Session cluster* deployment in Kubernetes has at least three components:
-## Deploy Flink cluster on Kubernetes
+* a *Deployment* which runs a [JobManager]({% link concepts/glossary.md %}#flink-jobmanager)
+* a *Deployment* for a pool of [TaskManagers]({% link concepts/glossary.md %}#flink-taskmanager)
+* a *Service* exposing the *JobManager's* REST and UI ports
-Using [the common resource definitions](#common-cluster-resource-definitions), launch the common cluster components
-with the `kubectl` command:
+Using the file contents provided in the [the common resource definitions](#common-cluster-resource-definitions), create the following files, and create the respective components with the `kubectl` command:
```sh
+ # Configuration and service definition
kubectl create -f flink-configuration-configmap.yaml
kubectl create -f jobmanager-service.yaml
+ # Create the deployments for the cluster
+ kubectl create -f jobmanager-session-deployment.yaml
+ kubectl create -f taskmanager-session-deployment.yaml
```
-Note that you could define your own customized options of `flink-conf.yaml` within `flink-configuration-configmap.yaml`.
-
-Then launch the specific components depending on whether you want to deploy a [Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster.
-
-You can then access the Flink UI via different ways:
+You can then access the Flink UI and submit jobs via different ways:
Review comment:
What about sticking to one (the easiest; I'd say `port-forward`) and listing the others in the references section? To shorten the Getting Started guide.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+cd; pwd
{% endhighlight %}
-#### Adding a TaskManager
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+export SHELLOPTS
+set -o igncr
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+Save the file and open a new bash shell.
-## High-Availability with Standalone
+### Setting up High-Availability
In order to enable HA for a standalone cluster, you have to use the [ZooKeeper HA services]({% link deployment/ha/zookeeper_ha.md %}).
Additionally, you have to configure your cluster to start multiple JobManagers.
-### Masters File (masters)
-
In order to start an HA-cluster configure the *masters* file in `conf/masters`:
- **masters file**: The *masters file* contains all hosts, on which JobManagers are started, and the ports to which the web user interface binds.
<pre>
-jobManagerAddress1:webUIPort1
+master1:webUIPort1
[...]
-jobManagerAddressX:webUIPortX
+masterX:webUIPortX
</pre>
By default, the job manager will pick a *random port* for inter process communication. You can change this via the [high-availability.jobmanager.port]({% link deployment/config.md %}#high-availability-jobmanager-port) key. This key accepts single ports (e.g. `50010`), ranges (`50000-50025`), or a combination of both (`50010,50011,50020-50025,50050-50075`).
-### Example: Standalone Cluster with 2 JobManagers
+**Example: Standalone Cluster with 2 JobManagers**
Review comment:
We could make this a subsection as well.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -252,7 +278,7 @@ To provide a custom location for the Flink configuration files, you can
<span class="label label-warning">Warning!</span> The mounted volume must contain all necessary configuration files.
The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases.
-### Using plugins
+### Using filesystem plugins
As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: in order to use plugins they must be
Review comment:
```suggestion
As described in the [plugins]({% link deployment/filesystems/plugins.md %}) documentation page: In order to use plugins they must be
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -344,64 +389,36 @@ as described in [how to run the Flink image](#how-to-run-flink-image).
ENV VAR_NAME value
```
- **Commands for building**:
+ **Commands for building**:
```sh
docker build -t custom_flink_image .
# optional push to your docker image registry if you have it,
# e.g. to distribute the custom image to your cluster
docker push custom_flink_image
```
-
-### Enabling Python
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Application cluster](#app-cluster-yml)
+ * [Session cluster](#session-cluster-yml)
Review comment:
I just realized that you used the collapsible code blocks. Reading through it right now, I guess, I prefer the Appendix approach we use in the Kubernetes documentation page.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
Review comment:
What about running this job in detached mode and adding the `--output` variable. This way we could show how a job is submitted having job-related parameters.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
Review comment:
We agreed on adding `$` as a prefix to each command. Please review all lines containing commands accordingly.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
Review comment:
```suggestion
We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a single TaskManager.
```
nit making it more explicit
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
Review comment:
```suggestion
## Deployment Modes
```
We have to add `Per-Job Mode` as well.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
Review comment:
```suggestion
This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distributed standalone cluster, which we describe in the [reference section](#the-start-and-stop-scripts).
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
Review comment:
```suggestion
Local deployment in Session Mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
Review comment:
```suggestion
Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level.
```
That's not true anymore. We enabled dynamic log4j config file scanning recently. But it's good enough to refer to the logging documentation here, I guess.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
Review comment:
Does it make sense to mention that log files of previous runs are not overwritten but appended by numbers. The most recent run has no suffix.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
Review comment:
```suggestion
### The Start and Stop Scripts
```
I would suggest this formatting following the way the other headlines are formatted. Not sure about `Start` and `Stop`, though
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
Review comment:
Considering `### Component Management Scripts` (or something similar), I would rename this section into `Starting and Stopping a Cluster` as we're also talking about `./bin/stop-cluster.sh` here.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+cd; pwd
{% endhighlight %}
-#### Adding a TaskManager
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+export SHELLOPTS
+set -o igncr
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+Save the file and open a new bash shell.
-## High-Availability with Standalone
+### Setting up High-Availability
In order to enable HA for a standalone cluster, you have to use the [ZooKeeper HA services]({% link deployment/ha/zookeeper_ha.md %}).
Additionally, you have to configure your cluster to start multiple JobManagers.
-### Masters File (masters)
-
In order to start an HA-cluster configure the *masters* file in `conf/masters`:
- **masters file**: The *masters file* contains all hosts, on which JobManagers are started, and the ports to which the web user interface binds.
<pre>
-jobManagerAddress1:webUIPort1
+master1:webUIPort1
[...]
-jobManagerAddressX:webUIPortX
+masterX:webUIPortX
</pre>
By default, the job manager will pick a *random port* for inter process communication. You can change this via the [high-availability.jobmanager.port]({% link deployment/config.md %}#high-availability-jobmanager-port) key. This key accepts single ports (e.g. `50010`), ranges (`50000-50025`), or a combination of both (`50010,50011,50020-50025,50050-50075`).
-### Example: Standalone Cluster with 2 JobManagers
+**Example: Standalone Cluster with 2 JobManagers**
1. **Configure high availability mode and ZooKeeper quorum** in `conf/flink-conf.yaml`:
Review comment:
The commands below are missing a leading `./`, e.g.: `$ bin/start-zookeeper-quorum.sh` -> `$ ./bin/start-zookeeper-quorum.sh`
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
Review comment:
We could refer to this specific example in the intro of the Getting Started section if we make these examples actual sections.
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
Review comment:
```suggestion
Note that the configuration key [jobmanager.rpc.address]({% link deployment/config.md %}#jobmanager-rpc-address) needs to be set to `master1` for this to work.
```
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -23,45 +23,53 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a *Flink Job* and *Session cluster* on [Kubernetes](https://kubernetes.io).
-
* This will be replaced by the TOC
{:toc}
-{% info %} This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes.
-You can find more information on native Kubernetes deployments [here]({% link deployment/resource-providers/native_kubernetes.md %}).
-## Setup Kubernetes
+## Getting Started
+
+This *Getting Started* guide describes how to deploy a *Session cluster* on [Kubernetes](https://kubernetes.io).
+
+### Introduction
+
+This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes, using Flink's standalone deployment.
+We generally recommend new users to deploy Flink on Kubernetes using [native Kubernetes deployments]({% link deployment/resource-providers/native_kubernetes.md %}).
-Please follow [Kubernetes' setup guide](https://kubernetes.io/docs/setup/) in order to deploy a Kubernetes cluster.
-If you want to run Kubernetes locally, we recommend using [MiniKube](https://kubernetes.io/docs/setup/minikube/).
+### Preparation
+
+This guide expects a Kubernetes environment to be present. You can ensure that your Kubernetes setup is working by running a command like `kubectl get nodes`, which lists all connected Kubelets.
+
+If you want to run Kubernetes locally, we recommend using [MiniKube](https://minikube.sigs.k8s.io/docs/start/).
<div class="alert alert-info" markdown="span">
<strong>Note:</strong> If using MiniKube please make sure to execute `minikube ssh 'sudo ip link set docker0 promisc on'` before deploying a Flink cluster.
Otherwise Flink components are not able to self reference themselves through a Kubernetes service.
Review comment:
```suggestion
Otherwise Flink components are not able to reference themselves through a Kubernetes service.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
Review comment:
```suggestion
`bin/start-cluster.sh` and `bin/stop-cluster.sh` rely on `conf/masters` and `conf/workers` to determine the number of cluster component instances.
```
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -81,96 +89,95 @@ You can then access the Flink UI via different ways:
{% highlight bash %}./bin/flink run -m <public-node-ip>:<node-port> ./examples/streaming/WordCount.jar{% endhighlight %}
-You can also access the queryable state of TaskManager if you create a `NodePort` service for it:
- 1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service on taskmanager. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
- 2. Run `kubectl get svc flink-taskmanager-query-state` to know the `node-port` of this service. Then you can create the [QueryableStateClient(<public-node-ip>, <node-port>]({% link dev/stream/state/queryable_state.md %}#querying-state) to submit the state queries.
-In order to terminate the Flink cluster, delete the specific [Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster components
-and use `kubectl` to terminate the common components:
+You can tear down the cluster using the following commands:
```sh
kubectl delete -f jobmanager-service.yaml
kubectl delete -f flink-configuration-configmap.yaml
- # if created then also the rest service
- kubectl delete -f jobmanager-rest-service.yaml
- # if created then also the queryable state service
- kubectl delete -f taskmanager-query-state-service.yaml
+ kubectl delete -f taskmanager-session-deployment.yaml
+ kubectl delete -f jobmanager-session-deployment.yaml
```
-### Deploy Session Cluster
-
-A *Flink Session cluster* is executed as a long-running Kubernetes Deployment.
-Note that you can run multiple Flink jobs on a *Session cluster*.
-Each job needs to be submitted to the cluster after the cluster has been deployed.
-A *Flink Session cluster* deployment in Kubernetes has at least three components:
-
-* a *Deployment* which runs a [JobManager]({% link concepts/glossary.md %}#flink-jobmanager)
-* a *Deployment* for a pool of [TaskManagers]({% link concepts/glossary.md %}#flink-taskmanager)
-* a *Service* exposing the *JobManager's* REST and UI ports
-
-After creating [the common cluster components](#deploy-flink-cluster-on-kubernetes), use [the Session specific resource definitions](#session-cluster-resource-definitions)
-to launch the *Session cluster* with the `kubectl` command:
-
-```sh
- kubectl create -f jobmanager-session-deployment.yaml
- kubectl create -f taskmanager-session-deployment.yaml
-```
+{% top %}
-To terminate the *Session cluster*, these components can be deleted along with [the common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:
+## Deployment Modes
-```sh
- kubectl delete -f taskmanager-session-deployment.yaml
- kubectl delete -f jobmanager-session-deployment.yaml
-```
+### Deploy Application Cluster
-### Deploy Job Cluster
+A *Flink Application cluster* is a dedicated cluster which runs a single application.
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
-You can find more details [here](#start-a-job-cluster).
+A basic *Flink Application cluster* deployment in Kubernetes has three components:
-A basic *Flink Job cluster* deployment in Kubernetes has three components:
-
-* a *Job* which runs a *JobManager*
+* an *Application* which runs a *JobManager*
* a *Deployment* for a pool of *TaskManagers*
* a *Service* exposing the *JobManager's* REST and UI ports
-Check [the Job cluster specific resource definitions](#job-cluster-resource-definitions) and adjust them accordingly.
+Check [the Job cluster specific resource definitions](#application-cluster-resource-definitions) and adjust them accordingly:
Review comment:
```suggestion
Check [the Application cluster specific resource definitions](#application-cluster-resource-definitions) and adjust them accordingly:
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
Review comment:
```suggestion
If password-less SSH access to the listed machines is configured, and they share the same directory structure, the scripts also support starting and stopping instances remotely.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
Review comment:
```suggestion
This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running on Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
```
Not sure, though? 😇 ...just based on the sound of it: `running something on Windows` 🤔
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -23,45 +23,53 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a *Flink Job* and *Session cluster* on [Kubernetes](https://kubernetes.io).
-
* This will be replaced by the TOC
{:toc}
-{% info %} This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes.
-You can find more information on native Kubernetes deployments [here]({% link deployment/resource-providers/native_kubernetes.md %}).
-## Setup Kubernetes
+## Getting Started
+
+This *Getting Started* guide describes how to deploy a *Session cluster* on [Kubernetes](https://kubernetes.io).
+
+### Introduction
+
+This page describes deploying a [standalone]({% link deployment/resource-providers/standalone/index.md %}) Flink cluster on top of Kubernetes, using Flink's standalone deployment.
+We generally recommend new users to deploy Flink on Kubernetes using [native Kubernetes deployments]({% link deployment/resource-providers/native_kubernetes.md %}).
-Please follow [Kubernetes' setup guide](https://kubernetes.io/docs/setup/) in order to deploy a Kubernetes cluster.
-If you want to run Kubernetes locally, we recommend using [MiniKube](https://kubernetes.io/docs/setup/minikube/).
+### Preparation
+
+This guide expects a Kubernetes environment to be present. You can ensure that your Kubernetes setup is working by running a command like `kubectl get nodes`, which lists all connected Kubelets.
+
+If you want to run Kubernetes locally, we recommend using [MiniKube](https://minikube.sigs.k8s.io/docs/start/).
<div class="alert alert-info" markdown="span">
<strong>Note:</strong> If using MiniKube please make sure to execute `minikube ssh 'sudo ip link set docker0 promisc on'` before deploying a Flink cluster.
Otherwise Flink components are not able to self reference themselves through a Kubernetes service.
</div>
-## Flink Docker image
+### Starting a Kubernetes Cluster (Session Mode)
+
+A *Flink Session cluster* is executed as a long-running Kubernetes Deployment. You can run multiple Flink jobs on a *Session cluster*.
+Each job needs to be submitted to the cluster after the cluster has been deployed.
-Before deploying the Flink Kubernetes components, please read [the Flink Docker image documentation]({% link deployment/resource-providers/standalone/docker.md %}),
-[its tags]({% link deployment/resource-providers/standalone/docker.md %}#image-tags), [how to customize the Flink Docker image]({% link deployment/resource-providers/standalone/docker.md %}#customize-flink-image) and
-[enable plugins]({% link deployment/resource-providers/standalone/docker.md %}#using-plugins) to use the image in the Kubernetes definition files.
+A *Flink Session cluster* deployment in Kubernetes has at least three components:
-## Deploy Flink cluster on Kubernetes
+* a *Deployment* which runs a [JobManager]({% link concepts/glossary.md %}#flink-jobmanager)
+* a *Deployment* for a pool of [TaskManagers]({% link concepts/glossary.md %}#flink-taskmanager)
+* a *Service* exposing the *JobManager's* REST and UI ports
-Using [the common resource definitions](#common-cluster-resource-definitions), launch the common cluster components
-with the `kubectl` command:
+Using the file contents provided in the [the common resource definitions](#common-cluster-resource-definitions), create the following files, and create the respective components with the `kubectl` command:
```sh
+ # Configuration and service definition
kubectl create -f flink-configuration-configmap.yaml
Review comment:
Missing `$` prefix (applies to the whole page).
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-
-For example, you can use the following aliases:
+### Introduction
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
-
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %})) are described on separate pages.
-We recommend using [Docker Compose]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-compose) or [Docker Swarm]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
-### Start a Session Cluster
+### Starting a Session Cluster on Docker
-A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after the cluster has been deployed.
To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
```sh
FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
Review comment:
The leading `$ ` prefix is missing (this applies to all commands on this page).
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -81,96 +89,95 @@ You can then access the Flink UI via different ways:
{% highlight bash %}./bin/flink run -m <public-node-ip>:<node-port> ./examples/streaming/WordCount.jar{% endhighlight %}
-You can also access the queryable state of TaskManager if you create a `NodePort` service for it:
- 1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service on taskmanager. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
- 2. Run `kubectl get svc flink-taskmanager-query-state` to know the `node-port` of this service. Then you can create the [QueryableStateClient(<public-node-ip>, <node-port>]({% link dev/stream/state/queryable_state.md %}#querying-state) to submit the state queries.
-In order to terminate the Flink cluster, delete the specific [Session](#deploy-session-cluster) or [Job](#deploy-job-cluster) cluster components
-and use `kubectl` to terminate the common components:
+You can tear down the cluster using the following commands:
```sh
kubectl delete -f jobmanager-service.yaml
kubectl delete -f flink-configuration-configmap.yaml
- # if created then also the rest service
- kubectl delete -f jobmanager-rest-service.yaml
- # if created then also the queryable state service
- kubectl delete -f taskmanager-query-state-service.yaml
+ kubectl delete -f taskmanager-session-deployment.yaml
+ kubectl delete -f jobmanager-session-deployment.yaml
```
-### Deploy Session Cluster
-
-A *Flink Session cluster* is executed as a long-running Kubernetes Deployment.
-Note that you can run multiple Flink jobs on a *Session cluster*.
-Each job needs to be submitted to the cluster after the cluster has been deployed.
-A *Flink Session cluster* deployment in Kubernetes has at least three components:
-
-* a *Deployment* which runs a [JobManager]({% link concepts/glossary.md %}#flink-jobmanager)
-* a *Deployment* for a pool of [TaskManagers]({% link concepts/glossary.md %}#flink-taskmanager)
-* a *Service* exposing the *JobManager's* REST and UI ports
-
-After creating [the common cluster components](#deploy-flink-cluster-on-kubernetes), use [the Session specific resource definitions](#session-cluster-resource-definitions)
-to launch the *Session cluster* with the `kubectl` command:
-
-```sh
- kubectl create -f jobmanager-session-deployment.yaml
- kubectl create -f taskmanager-session-deployment.yaml
-```
+{% top %}
-To terminate the *Session cluster*, these components can be deleted along with [the common ones](#deploy-flink-cluster-on-kubernetes) with the `kubectl` command:
+## Deployment Modes
-```sh
- kubectl delete -f taskmanager-session-deployment.yaml
- kubectl delete -f jobmanager-session-deployment.yaml
-```
+### Deploy Application Cluster
-### Deploy Job Cluster
+A *Flink Application cluster* is a dedicated cluster which runs a single application.
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
-You can find more details [here](#start-a-job-cluster).
+A basic *Flink Application cluster* deployment in Kubernetes has three components:
-A basic *Flink Job cluster* deployment in Kubernetes has three components:
-
-* a *Job* which runs a *JobManager*
+* an *Application* which runs a *JobManager*
* a *Deployment* for a pool of *TaskManagers*
* a *Service* exposing the *JobManager's* REST and UI ports
-Check [the Job cluster specific resource definitions](#job-cluster-resource-definitions) and adjust them accordingly.
+Check [the Job cluster specific resource definitions](#application-cluster-resource-definitions) and adjust them accordingly:
The `args` attribute in the `jobmanager-job.yaml` has to specify the main class of the user job.
See also [how to specify the JobManager arguments]({% link deployment/resource-providers/standalone/docker.md %}#jobmanager-additional-command-line-arguments) to understand
how to pass other `args` to the Flink image in the `jobmanager-job.yaml`.
-The *job artifacts* should be available from the `job-artifacts-volume` in [the resource definition examples](#job-cluster-resource-definitions).
+The *job artifacts* should be available from the `job-artifacts-volume` in [the resource definition examples](#application-cluster-resource-definitions).
The definition examples mount the volume as a local directory of the host assuming that you create the components in a minikube cluster.
If you do not use a minikube cluster, you can use any other type of volume, available in your Kubernetes cluster, to supply the *job artifacts*.
-Alternatively, you can build [a custom image]({% link deployment/resource-providers/standalone/docker.md %}#start-a-job-cluster) which already contains the artifacts instead.
+Alternatively, you can build [a custom image]({% link deployment/resource-providers/standalone/docker.md %}#advanced-customization) which already contains the artifacts instead.
-After creating [the common cluster components](#deploy-flink-cluster-on-kubernetes), use [the Job cluster specific resource definitions](#job-cluster-resource-definitions)
-to launch the cluster with the `kubectl` command:
+After creating [the common cluster components](#common-cluster-resource-definitions), use [the Application cluster specific resource definitions](#application-cluster-resource-definitions) to launch the cluster with the `kubectl` command:
```sh
kubectl create -f jobmanager-job.yaml
kubectl create -f taskmanager-job-deployment.yaml
```
-To terminate the single job cluster, these components can be deleted along with [the common ones](#deploy-flink-cluster-on-kubernetes)
+To terminate the single application cluster, these components can be deleted along with [the common ones](#common-cluster-resource-definitions)
with the `kubectl` command:
```sh
kubectl delete -f taskmanager-job-deployment.yaml
kubectl delete -f jobmanager-job.yaml
```
-## High-Availability with Standalone Kubernetes
+### Per-Job Cluster Mode
+Flink on Standalone Kubernetes does not support the Per-Job Cluster Mode.
-For high availability on Kubernetes, you can use the [existing high availability services]({% link deployment/ha/index.md %}).
+### Session Mode
+
+Deployment of a Session cluster is explained in the [Getting Started](#getting-started) guide at the top of this page.
+
+{% top %}
+
+## Flink on Standalone Kubernetes Reference
+
+### Configuration
-### How to configure Kubernetes HA Services
+All configuration options are listed on the [configuration page]({% link deployment/config.md %}). Configuration options can be added to the `flink-conf.yaml` section of the `flink-configuration-configmap.yaml` config map.
-Session Mode, Per-Job Mode, and Application Mode clusters support using the Kubernetes high availability service. Users just need to add the following Flink config options to [flink-configuration-configmap.yaml](#common-cluster-resource-definitions). All other yamls do not need to be updated.
+### Debugging and Log Access
-<span class="label label-info">Note</span> The filesystem which corresponds to the scheme of your configured HA storage directory must be available to the runtime. Refer to [custom Flink image]({% link deployment/resource-providers/standalone/docker.md %}#customize-flink-image) and [enable plugins]({% link deployment/resource-providers/standalone/docker.md %}#using-plugins) for more information.
+Many common errors are easy to detect by checking Flink's log files. If you have access to Flink's web user interface, you can access the JobManager and TaskManager logs from there.
+
+If there are problems starting Flink, you can also use Kubernetes utilities to access the logs. Use `kubectl get pods` to see all running pods.
+For the quickstart example from above, you should see three pods:
+```
+$ kubectl get pods
+NAME READY STATUS RESTARTS AGE
+flink-jobmanager-589967dcfc-m49xv 1/1 Running 3 3m32s
+flink-taskmanager-64847444ff-7rdl4 1/1 Running 3 3m28s
+flink-taskmanager-64847444ff-nnd6m 1/1 Running 3 3m28s
+```
+
+You can now access the logs by running `kubectl logs flink-jobmanager-589967dcfc-m49xv`.
Review comment:
I would suggest removing the period at the end for the sake of not risking misinterpreting it as being part of the command.
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -190,6 +197,14 @@ data:
...
{% endhighlight %}
+### Enabling Queryable State
+
+You can access the queryable state of TaskManager if you create a `NodePort` service for it:
+ 1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service on taskmanager. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
Review comment:
```suggestion
1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service for the `taskmanager` pod. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
```
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -190,6 +197,14 @@ data:
...
{% endhighlight %}
+### Enabling Queryable State
+
+You can access the queryable state of TaskManager if you create a `NodePort` service for it:
+ 1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service on taskmanager. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
+ 2. Run `kubectl get svc flink-taskmanager-query-state` to know the `node-port` of this service. Then you can create the [QueryableStateClient(<public-node-ip>, <node-port>]({% link dev/stream/state/queryable_state.md %}#querying-state) to submit the state queries.
Review comment:
Is there a reason why we change the formatting of `NodePort` to `node-port`?
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
Review comment:
```suggestion
#### Start and Stop Flink Components
```
Considering the other headline change I proposed above, I would prefer having headlines without script names since we usually refer to those scripts including the path from within `FLINK_HOME`.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-
-For example, you can use the following aliases:
+### Introduction
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
-
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %})) are described on separate pages.
Review comment:
```suggestion
Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}), are described on separate pages.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-
-For example, you can use the following aliases:
+### Introduction
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
-
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %})) are described on separate pages.
-We recommend using [Docker Compose]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-compose) or [Docker Swarm]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
-### Start a Session Cluster
+### Starting a Session Cluster on Docker
-A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after the cluster has been deployed.
To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
Review comment:
```suggestion
To deploy a *Flink Session cluster* with Docker, you need to start a JobManager container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
```
We don't emphasize the Flink components.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
Review comment:
```suggestion
To shut down the cluster, either terminate (e.g. with `CTRL-C`) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+cd; pwd
{% endhighlight %}
-#### Adding a TaskManager
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+export SHELLOPTS
+set -o igncr
{% endhighlight %}
-Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.
+Save the file and open a new bash shell.
Review comment:
```suggestion
4. Re-initialize your shell to apply the changes.
```bash
source .bash_profile
```
```
This want, I didn't test, though, due to the lack of having Cygwin/Windows available. 😇
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
Review comment:
```suggestion
This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster using Docker containers.
```
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+cd; pwd
{% endhighlight %}
-#### Adding a TaskManager
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
{% highlight bash %}
-bin/taskmanager.sh start|start-foreground|stop|stop-all
+export SHELLOPTS
+set -o igncr
{% endhighlight %}
Review comment:
```
2. Determine your home directory based on the Cygwin root path.
```bash
cd; pwd
```
3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following (if the file does not exist you will have to create it):
```bash
export SHELLOPTS
set -o igncr
```
```
This fixes the enumeration issue (the old version counted 1. 2. 1.). The code blocks need to have exactly 4 spaces according to the comments in [Jekyll issue #588](https://github.com/jekyll/jekyll/issues/588) (I didn't test other settings).
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
-Assuming that you are on the master node and inside the Flink directory:
+#### start-cluster.sh
+The scripts provided with the standalone mode (in the `bin/` directory) use the `conf/workers` and `conf/masters` files, to determine the number of cluster instances to start and stop with the `bin/start-cluster.sh` and `bin/stop-cluster.sh` scripts.
+
+If password-less ssh access to the listed machines is configured, and they share the same directory structure, the script also supports starting and stopping instances remotely.
+
+**Example 1: Start a cluster with 2 TaskManagers locally**
+
+`conf/masters` contents:
{% highlight bash %}
-bin/start-cluster.sh
+localhost
{% endhighlight %}
-To stop Flink, there is also a `stop-cluster.sh` script.
+`conf/workers` contents:
+{% highlight bash %}
+localhost
+localhost
+{% endhighlight %}
-{% top %}
+**Example 2: Start a distributed cluster JobMangers**
+
+This assumes a cluster with 4 machines (`master1, worker1, worker2, worker3`), which all can reach each other over the network.
+
+`conf/masters` contents:
+{% highlight bash %}
+master1
+{% endhighlight %}
+
+`conf/workers` contents:
+{% highlight bash %}
+worker1
+worker2
+worker3
+{% endhighlight %}
-### Adding JobManager/TaskManager Instances to a Cluster
+Note that the configuration key `jobmanager.rpc.address` needs to be set to `master1` for this to work.
-You can add both JobManager and TaskManager instances to your running cluster with the `bin/jobmanager.sh` and `bin/taskmanager.sh` scripts.
+We show a third example with a standby JobManager in the [high-availability section](#setting-up-high-availability).
-#### Adding a JobManager
+#### (jobmanager|taskmanager).sh
+
+The `bin/jobmanager.sh` and `bin/taskmanager.sh` script support starting the respective daemon in the background (using the `start` argument), or in the foreground (using `start-foreground`). In the foreground mode, the logs are printed to standard out. This mode is useful for deployment scenarios where another process is controlling the Flink daemon (e.g. Docker).
+
+The scripts can be called multiple times, for example if multiple TaskManagers are needed. The instances are tracked by the scripts, and can be stopped one-by-one (using `stop`) or all together (using `stop-all`).
+
+#### Windows Cygwin Users
+
+If you are installing Flink from the git repository and you are using the Windows git shell, Cygwin can produce a failure similar to this one:
+
+{% highlight bash %}
+c:/flink/bin/start-cluster.sh: line 30: $'\r': command not found
+{% endhighlight %}
+
+This error occurs because git is automatically transforming UNIX line endings to Windows style line endings when running in Windows. The problem is that Cygwin can only deal with UNIX style line endings. The solution is to adjust the Cygwin settings to deal with the correct line endings by following these three steps:
+
+1. Start a Cygwin shell.
+
+2. Determine your home directory by entering
{% highlight bash %}
-bin/jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all
+cd; pwd
{% endhighlight %}
-#### Adding a TaskManager
+ This will return a path under the Cygwin root path.
+
+3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following: (If the file does not exist you will have to create it)
Review comment:
```suggestion
3. Using NotePad, WordPad or a different text editor open the file `.bash_profile` in the home directory and append the following (if the file does not exist you will have to create it):
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-
-For example, you can use the following aliases:
+### Introduction
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
-
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
Review comment:
```suggestion
You can use the Docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
+
+This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
Review comment:
```suggestion
This allows you to deploy a standalone cluster (Session or Per-Job Mode) in any containerised environment, for example:
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
+
+This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
+* manually in a local Docker setup,
+* [in a (native) Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
+and deploys *TaskManagers* on demand so that you do not have to do it manually.
+
+The next chapters describe how to start a single Flink Docker container for various purposes.
+
+Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+
+We recommend using [Docker Compose](#flink-with-docker-compose) or [Docker Swarm](#flink-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
Review comment:
```suggestion
We recommend using [Docker Compose](#flink-with-docker-compose) or [Docker Swarm](#flink-with-docker-swarm) for deploying Flink in Session Mode to ease system configuration.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
+
+This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
+* manually in a local Docker setup,
+* [in a (native) Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
+and deploys *TaskManagers* on demand so that you do not have to do it manually.
+
+The next chapters describe how to start a single Flink Docker container for various purposes.
+
+Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+
+We recommend using [Docker Compose](#flink-with-docker-compose) or [Docker Swarm](#flink-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
+
+
+### Application Mode on Docker
+
+A *Flink Application cluster* is a dedicated cluster which runs a single job. This is sometimes also called a *Per-Job cluster*.
Review comment:
```suggestion
A *Flink Application cluster* is a dedicated cluster which runs a single job.
```
I'm afraid that claiming the Application cluster and Per-Job cluster being the same thing can be a bit confusing as we differentiate between them in the Deployment Mode section.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
Review comment:
Not sure whether we need to refer to the glossary here since we don't do it anywhere else...
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,12 +198,44 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
+### Per-Job Cluster Mode
+
+Per-Job Cluster Mode is not supported by Docker.
+
+### Session Mode on Docker
+
+Local deployment in the session mode has already been described in the [introduction](#starting-a-session-cluster-on-docker) above.
+
+
+{% top %}
+
+## Flink on Docker Reference
+
+### Image tags
+
+The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on Docker Hub and serves images of Flink version 1.2.1 and later.
+The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+
+Images for each supported combination of Flink and Scala versions are available, and
+[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
+
+For example, you can use the following aliases:
+
+* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
+* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-When you run the Flink containers, there may be a need to customize them.
-The next chapters describe some how-tos of what you can usually customize.
+<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
+versions (for example `flink:1.11-scala_2.12`).
+This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
+from the versions provided by the docker image.
+
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
+Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
+that do not include a bundled Hadoop distribution.
-### Configure options
+
+### Passing configuration via environment variables
When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`:
Review comment:
I suggest renaming the `FLINK_PROPERTIES` env variable into something else (e.g. `FLINK_CONFIG`) to not confuse it with the Docker environment variable used in the command.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -268,6 +294,27 @@ The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names sep
There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
+### Enabling Python
+
+To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
Review comment:
```suggestion
To build a custom image which has Python and PyFlink prepared, you can refer to the following Dockerfile:
```
##########
File path: docs/deployment/cli.zh.md
##########
@@ -35,7 +35,7 @@ It connects to the running JobManager specified in `conf/flink-config.yaml`.
A prerequisite for the commands listed in this section to work is to have a running Flink deployment
like [Kubernetes]({% link deployment/resource-providers/native_kubernetes.zh.md %}),
[YARN]({% link deployment/resource-providers/yarn.zh.md %}) or any other option available. Feel free to
-[start a Flink cluster locally]({% link deployment/resource-providers/standalone/local.zh.md %}#start-a-local-flink-cluster)
+[start a Flink cluster locally]({% link deployment/resource-providers/standalone/index.zh.md %}#start-a-local-flink-cluster)
Review comment:
```suggestion
[start a Flink cluster locally]({% link deployment/resource-providers/standalone/index.zh.md %}#starting-a-standalone-cluster-session-mode)
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -288,13 +335,11 @@ There are several ways in which you can further customize the Flink image:
* add other libraries to `/opt/flink/lib` (e.g. Hadoop)
* add other plugins to `/opt/flink/plugins`
-See also: [How to provide dependencies in the classpath]({% link index.md %}#how-to-provide-dependencies-in-the-classpath).
-
You can customize the Flink image in several ways:
* **override the container entry point** with a custom script where you can run any bootstrap actions.
-At the end you can call the standard `/docker-entrypoint.sh` script of the Flink image with the same arguments
-as described in [how to run the Flink image](#how-to-run-flink-image).
+ At the end you can call the standard `/docker-entrypoint.sh` script of the Flink image with the same arguments
+ as described in [supported deployment modes](#deployment-modes).
The following example creates a custom entry point script which enables more libraries and plugins.
The custom script, custom library and plugin are provided from a mounted volume.
Review comment:
The linewraps look weird with the comments being on the same line as the commands in the code below. Can't we move the comments in front of the command on a separate line to avoid having linewraps?
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-
-For example, you can use the following aliases:
+### Introduction
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
-
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %})) are described on separate pages.
-We recommend using [Docker Compose]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-compose) or [Docker Swarm]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
-### Start a Session Cluster
+### Starting a Session Cluster on Docker
-A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after the cluster has been deployed.
To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
Review comment:
This applies to other occurrences on this page, too.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -344,64 +389,36 @@ as described in [how to run the Flink image](#how-to-run-flink-image).
ENV VAR_NAME value
```
- **Commands for building**:
+ **Commands for building**:
```sh
docker build -t custom_flink_image .
# optional push to your docker image registry if you have it,
# e.g. to distribute the custom image to your cluster
docker push custom_flink_image
```
-
-### Enabling Python
-To build a custom image which has Python and Pyflink prepared, you can refer to the following Dockerfile:
-{% highlight Dockerfile %}
-FROM flink
-
-# install python3 and pip3
-RUN apt-get update -y && \
-apt-get install -y python3.7 python3-pip python3.7-dev && rm -rf /var/lib/apt/lists/*
-RUN ln -s /usr/bin/python3 /usr/bin/python
-# install Python Flink
-RUN pip3 install apache-flink
-{% endhighlight %}
-
-Build the image named as **pyflink:latest**:
-
-{% highlight bash %}
-sudo docker build -t pyflink:latest .
-{% endhighlight %}
-
-{% top %}
-
-## Flink with Docker Compose
+### Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally.
-The next chapters show examples of configuration files to run Flink.
+The next sections show examples of configuration files to run Flink.
-### Usage
+#### Usage
* Create the `yaml` files with the container configuration, check examples for:
- * [Session cluster](#session-cluster-with-docker-compose)
- * [Job cluster](#job-cluster-with-docker-compose)
+ * [Application cluster](#app-cluster-yml)
+ * [Session cluster](#session-cluster-yml)
Review comment:
I couldn't find the corresponding anchor points/yaml files. The targets of these links do not exist.
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -190,6 +197,14 @@ data:
...
{% endhighlight %}
+### Enabling Queryable State
+
+You can access the queryable state of TaskManager if you create a `NodePort` service for it:
+ 1. Run `kubectl create -f taskmanager-query-state-service.yaml` to create the `NodePort` service on taskmanager. The example of `taskmanager-query-state-service.yaml` can be found in [appendix](#common-cluster-resource-definitions).
+ 2. Run `kubectl get svc flink-taskmanager-query-state` to know the `node-port` of this service. Then you can create the [QueryableStateClient(<public-node-ip>, <node-port>]({% link dev/stream/state/queryable_state.md %}#querying-state) to submit the state queries.
Review comment:
Ah, I see... because we're referring to the actual port. I'd suggest writing `<node-port>` then.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -124,13 +118,13 @@ The *job artifacts* are included into the class path of Flink's JVM process with
To deploy a cluster for a single job with Docker, you need to
* make *job artifacts* available locally *in all containers* under `/opt/flink/usrlib`,
-* start a *JobManager* container in the *Job Cluster* mode
+* start a *JobManager* container in the *Application Cluster* mode
Review comment:
The other times, we format it as "Application cluster". Additionally, there are two many formattings here, in my opinion.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -178,7 +172,7 @@ the *JobManager* and *TaskManagers*:
docker run flink_with_job_artifacts taskmanager
```
-The `standalone-job` argument starts a *JobManager* container in the *Job Cluster* mode.
+The `standalone-job` argument starts a *JobManager* container in the *Standalone Application Cluster* mode.
Review comment:
```suggestion
The `standalone-job` argument starts a JobManager container in the Application Mode.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
Review comment:
```suggestion
Submission of a job is now possible like this (assuming you have a local distribution of Flink available):
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -23,66 +23,26 @@ specific language governing permissions and limitations
under the License.
-->
-[Docker](https://www.docker.com) is a popular container runtime.
-There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
-You can use the docker images to deploy a *Session* or *Job cluster* in a containerized environment, e.g.,
-[standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}).
-
* This will be replaced by the TOC
{:toc}
-## Docker Hub Flink images
-The [Flink Docker repository](https://hub.docker.com/_/flink/) is hosted on
-Docker Hub and serves images of Flink version 1.2.1 and later.
-The source for these images can be found in the [Apache flink-docker](https://github.com/apache/flink-docker) repository.
+## Getting Started
-### Image tags
+This *Getting Started* section guides you through the local setup (on one machine, but in separate containers) of a Flink cluster.
-Images for each supported combination of Flink and Scala versions are available, and
-[tag aliases](https://hub.docker.com/_/flink?tab=tags) are provided for convenience.
-
-For example, you can use the following aliases:
+### Introduction
-* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
-* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-
-<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
-versions (for example `flink:1.11-scala_2.12`).
-This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
-from the versions provided by the docker image.
-
-<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
-You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
-Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
-that do not include a bundled Hadoop distribution.
-
-## How to run a Flink image
-
-The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
-You can run its entry point in the following modes:
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#start-a-session-cluster)
-* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Job cluster](#start-a-job-cluster)
-* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
-
-This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
-* manually in a local Docker setup,
-* [in a Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
-* [with Docker Compose](#flink-with-docker-compose),
-* [with Docker swarm](#flink-with-docker-swarm).
-
-<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
-and deploys *TaskManagers* on demand so that you do not have to do it manually.
-
-The next chapters describe how to start a single Flink Docker container for various purposes.
+[Docker](https://www.docker.com) is a popular container runtime.
+There are Docker images for Apache Flink available [on Docker Hub](https://hub.docker.com/_/flink).
+You can use the docker images to deploy a *Session* or *Application cluster* on Docker. This page focuses on the setup of Flink on Docker, Docker Swarm and Docker compose.
-Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+Deployment into managed containerized environments, such as [standalone Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}) or [native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %})) are described on separate pages.
-We recommend using [Docker Compose]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-compose) or [Docker Swarm]({% link deployment/resource-providers/standalone/docker.md %}#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
-### Start a Session Cluster
+### Starting a Session Cluster on Docker
-A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after the cluster has been deployed.
To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
```sh
FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
Review comment:
Just for the record, we agreed on having `$ ` as a prefix for each command, use long parameter names, have one parameter per line for commands that have more than one parameter and have a 6-spaces indentation per parameter line.
##########
File path: docs/deployment/cli.md
##########
@@ -36,7 +36,7 @@ It connects to the running JobManager specified in `conf/flink-config.yaml`.
A prerequisite for the commands listed in this section to work is to have a running Flink deployment
like [Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}),
[YARN]({% link deployment/resource-providers/yarn.md %}) or any other option available. Feel free to
-[start a Flink cluster locally]({% link deployment/resource-providers/standalone/local.md %}#start-a-local-flink-cluster)
+[start a Flink cluster locally]({% link deployment/resource-providers/standalone/index.md %}#start-a-local-flink-cluster)
Review comment:
```suggestion
[start a Flink cluster locally]({% link deployment/resource-providers/standalone/index.md %}#starting-a-standalone-cluster-session-mode)
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -268,6 +294,27 @@ The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names sep
Review comment:
Would it make sense to mention that no path information is needed. Only the basic filename of the plugin jar as it can be found in the `opt/` folder of the Flink distribution needs to be provided.
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -204,12 +198,44 @@ You can provide the following additional command line arguments to the cluster e
If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command.
-## Customize Flink image
+### Per-Job Cluster Mode
+
+Per-Job Cluster Mode is not supported by Docker.
+
+### Session Mode on Docker
+
+Local deployment in the session mode has already been described in the [introduction](#starting-a-session-cluster-on-docker) above.
Review comment:
```suggestion
Local deployment in the Session Mode has already been described in the [Getting Started](#starting-a-session-cluster-on-docker) section above.
```
##########
File path: docs/deployment/resource-providers/standalone/docker.md
##########
@@ -112,10 +72,44 @@ docker run \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+The web interface is now available at [localhost:8081](http://localhost:8081).
-### Start a Job Cluster
-A *Flink Job cluster* is a dedicated cluster which runs a single job.
+Submission of a job is now possible like this (assuming you have a local distribution of Flink available)
+
+```sh
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+To shut down the cluster, either terminate (e.g. with CTRL-C) the JobManager and TaskManager processes, or use `docker ps` to identify and `docker stop` to terminate the containers.
+
+## Deployment Modes
+
+The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
+You can run its entry point in the following modes:
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Session cluster](#starting-a-session-cluster-on-docker)
+* [JobManager]({% link concepts/glossary.md %}#flink-jobmanager) for [a Application cluster](#application-mode-on-docker)
+* [TaskManager]({% link concepts/glossary.md %}#flink-taskmanager) for any cluster
+
+This allows you to deploy a standalone cluster (Session or Job) in any containerised environment, for example:
+* manually in a local Docker setup,
+* [in a (native) Kubernetes cluster]({% link deployment/resource-providers/standalone/kubernetes.md %}),
+* [with Docker Compose](#flink-with-docker-compose),
+* [with Docker swarm](#flink-with-docker-swarm).
+
+<span class="label label-info">Note</span> [The native Kubernetes]({% link deployment/resource-providers/native_kubernetes.md %}) also runs the same image by default
+and deploys *TaskManagers* on demand so that you do not have to do it manually.
+
+The next chapters describe how to start a single Flink Docker container for various purposes.
+
+Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+
+We recommend using [Docker Compose](#flink-with-docker-compose) or [Docker Swarm](#flink-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
Review comment:
Alternatively, you can also still stick to `Flink cluster`. My main motivation was to remove the capitalized `C`. 8)
##########
File path: docs/deployment/resource-providers/standalone/index.md
##########
@@ -24,153 +24,203 @@ specific language governing permissions and limitations
under the License.
-->
-This page provides instructions on how to run Flink in a *fully distributed fashion* on a *static* (but possibly heterogeneous) cluster.
-
* This will be replaced by the TOC
{:toc}
-## Requirements
-### Software Requirements
+## Getting Started
-Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows) and expects the cluster to consist of **one master node** and **one or more worker nodes**. Before you start to setup the system, make sure you have the following software installed **on each node**:
+This *Getting Started* section guides you through the local setup (on one machine, but in separate processes) of a Flink cluster. This can easily be expanded to set up a distibuted standalone cluster, which we describe in the [reference section](#distributed-cluster-setup).
-- **Java 1.8.x** or higher,
-- **ssh** (sshd must be running to use the Flink scripts that manage
- remote components)
+### Introduction
-If your cluster does not fulfill these software requirements you will need to install/upgrade it.
+The standalone mode is the most barebone way of deploying Flink: The Flink services described in the [deployment overview]({% link deployment/index.md %}) are just launched as processes on the operating system. Unlike deploying Flink with a resource provider such as Kubernetes or YARN, you have to take care of restarting failed processes, or allocation and de-allocation of resources during operation.
-Having __passwordless SSH__ and
-__the same directory structure__ on all your cluster nodes will allow you to use our scripts to control
-everything.
+In the additional subpages of the standalone mode resource provider, we describe additional deployment methods which are based on the standalone mode: [Deployment in Docker containers]({% link deployment/resource-providers/standalone/docker.md %}), and on [Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %}).
-{% top %}
+### Preparation
-### `JAVA_HOME` Configuration
+Flink runs on all *UNIX-like environments*, e.g. **Linux**, **Mac OS X**, and **Cygwin** (for Windows). Before you start to setup the system, make sure you have the fulfilled the following requirements.
-Flink requires the `JAVA_HOME` environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.
+- **Java 1.8.x** or higher installed,
+- Downloaded a recent Flink distribution from the [download page]({{ site.download_url }}) and unpacked it.
-You can set this variable in `conf/flink-conf.yaml` via the `env.java.home` key.
+### Starting a Standalone Cluster (Session Mode)
-{% top %}
+These steps show how to launch a Flink standalone cluster, and submit an example job:
+
+{% highlight bash %}
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (1) Start Cluster
+./bin/start-cluster.sh
+
+# (2) You can now access the Flink Web Interface on http://localhost:8081
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop the cluster again
+./bin/stop-cluster.sh
+{% endhighlight %}
+
+In step `(1)`, we've started 2 processes: A JVM for the JobManager, and a JVM for the TaskManager. The JobManager is serving the web interface accessible at [localhost:8081](http://localhost:8081).
+In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that submits an application to the JobManager.
+
+## Deployment Modes Supported by the Standalone Cluster
+
+### Application Mode
-## Flink Setup
+To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script.
+We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a TaskManager.
-Go to the [downloads page]({{ site.download_url }}) and get the ready-to-run package.
-After downloading the latest release, copy the archive to your master node and extract it:
+The application jar file needs to be available in the classpath. The easiest approach to achieve that is putting the jar into the `lib/` folder:
{% highlight bash %}
-tar xzf flink-*.tgz
-cd flink-*
+cp ./examples/streaming/TopSpeedWindowing.jar lib/
{% endhighlight %}
-### Configuring Flink
+Then, we can launch the JobManager:
-After having extracted the system files, you need to configure Flink for the cluster by editing *conf/flink-conf.yaml*.
+{% highlight bash %}
+./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
+{% endhighlight %}
-Set the `jobmanager.rpc.address` key to point to your master node. You should also define the maximum amount of main memory Flink is allowed to allocate on each node by setting the `jobmanager.memory.process.size` and `taskmanager.memory.process.size` keys.
+The web interface is now available at [localhost:8081](http://localhost:8081). However, the application won't be able to start, because there are no TaskManagers running yet:
-These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting `taskmanager.memory.process.size` or `taskmanager.memory.flink.size` in *conf/flink-conf.yaml* on those specific nodes.
+{% highlight bash %}
+./bin/taskmanager.sh start
+{% endhighlight %}
-Finally, you must provide a list of all nodes in your cluster that shall be used as worker nodes, i.e., nodes running a TaskManager. Edit the file *conf/workers* and enter the IP/host name of each worker node.
+Note: You can start multiple TaskManagers, if your application needs more resources.
-The following example illustrates the setup with three nodes (with IP addresses from _10.0.0.1_
-to _10.0.0.3_ and hostnames _master_, _worker1_, _worker2_) and shows the contents of the
-configuration files (which need to be accessible at the same path on all machines):
+Stopping the services is also supported via the scripts:
-<div class="row">
- <div class="col-md-6 text-center">
- <img src="{% link /page/img/quickstart_cluster.png %}" style="width: 60%">
- </div>
-<div class="col-md-6">
- <div class="row">
- <p class="lead text-center">
- /path/to/<strong>flink/conf/<br>flink-conf.yaml</strong>
- <pre>jobmanager.rpc.address: 10.0.0.1</pre>
- </p>
- </div>
-<div class="row" style="margin-top: 1em;">
- <p class="lead text-center">
- /path/to/<strong>flink/<br>conf/workers</strong>
- <pre>
-10.0.0.2
-10.0.0.3</pre>
- </p>
-</div>
-</div>
-</div>
+{% highlight bash %}
+./bin/taskmanager.sh stop
+./bin/standalone-job.sh stop
+{% endhighlight %}
-The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.
-Please see the [configuration page]({% link deployment/config.md %}) for details and additional configuration options.
+### Session Mode
-In particular,
+Local deployment in the session mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above.
- * the amount of available memory per JobManager (`jobmanager.memory.process.size`),
- * the amount of available memory per TaskManager (`taskmanager.memory.process.size` and check [memory setup guide]({% link deployment/memory/mem_tuning.md %}#configure-memory-for-standalone-deployment)),
- * the number of available CPUs per machine (`taskmanager.numberOfTaskSlots`),
- * the total number of CPUs in the cluster (`parallelism.default`) and
- * the temporary directories (`io.tmp.dirs`)
+## Standalone Cluster Reference
-are very important configuration values.
+### Configuration
-{% top %}
+All available configuration options are listed on the [configuration page]({% link deployment/config.md %}), in particular the [Basic Setup]({% link deployment/config.md %}#basic-setup) section contains good advise on configuring the ports, memory, parallelism etc.
+
+### Debugging
+
+If Flink is behaving unexpectedly, we recommend looking at Flink's log files as a starting point for further investigations.
+
+The log files are located in the `logs/` directory. There's a `.log` file for each Flink service running on this machine.
+
+Alternatively, logs are available from the Flink web frontend (both for the JobManager and each TaskManager).
+
+By default, Flink is logging on the "INFO" log level, which provides basic information for all obvious issues. For cases where Flink supposedly behaving wrongly, reducing the log level to "DEBUG" is advised. The logging level is controlled via the `conf/log4.properties` file.
+Setting `rootLogger.level = DEBUG` will boostrap Flink on the DEBUG log level. Note that a restart of Flink is required for the changes to take effect.
-### Starting Flink
+There's a dedicated page on the [logging]({%link deployment/advanced/logging.md %}) in Flink.
-The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the *workers* file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.
+### The start and stop scripts
Review comment:
Alternatively, what about `### Component Management Scripts`? Since we're also referring to `bin/jobmanager.sh` and `bin/taskmanager.sh`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger closed pull request #14346:
URL: https://github.com/apache/flink/pull/14346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #14346:
URL: https://github.com/apache/flink/pull/14346#discussion_r540260761
##########
File path: docs/deployment/resource-providers/standalone/kubernetes.md
##########
@@ -23,7 +23,7 @@ specific language governing permissions and limitations
under the License.
-->
-This page describes how to deploy a *Flink Job* and *Session cluster* on [Kubernetes](https://kubernetes.io).
+This page describes how to deploy a *Flink application cluster* and *Session cluster* on [Kubernetes](https://kubernetes.io).
Review comment:
Okay, I'll give it a try
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ad8d2e75d7f717ecc0f8ec1f29189306f50a214 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811)
* 70b61ad7df7f89069a7f3b0a20445e709ce91ac8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14346: [FLINK-20354] Rework standalone docs pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14346:
URL: https://github.com/apache/flink/pull/14346#issuecomment-741680007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10708",
"triggerID" : "1feff43f3fba39641c0403e7968dfca592b400d9",
"triggerType" : "PUSH"
}, {
"hash" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10811",
"triggerID" : "2ad8d2e75d7f717ecc0f8ec1f29189306f50a214",
"triggerType" : "PUSH"
}, {
"hash" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841",
"triggerID" : "70b61ad7df7f89069a7f3b0a20445e709ce91ac8",
"triggerType" : "PUSH"
}, {
"hash" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847",
"triggerID" : "9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70b61ad7df7f89069a7f3b0a20445e709ce91ac8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10841)
* 9ca5fa15b0fd8c3ca8d76672866c6d8db2623be8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10847)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org