You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by rm...@apache.org on 2020/06/05 11:04:20 UTC
[flink] branch release-1.11 updated: [FLINK-17974][docs] Extend
docker documentation
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch release-1.11
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.11 by this push:
new b20ea34 [FLINK-17974][docs] Extend docker documentation
b20ea34 is described below
commit b20ea34d3c10254a82e46875419cd98d2986085e
Author: Robert Metzger <rm...@apache.org>
AuthorDate: Thu Jun 4 14:41:53 2020 +0200
[FLINK-17974][docs] Extend docker documentation
This closes #12490
---
docs/ops/deployment/docker.md | 59 +++++++++++++++++++++++++++++---------
docs/ops/deployment/docker.zh.md | 61 ++++++++++++++++++++++++++++++----------
2 files changed, 91 insertions(+), 29 deletions(-)
diff --git a/docs/ops/deployment/docker.md b/docs/ops/deployment/docker.md
index a1775c3..2525185 100644
--- a/docs/ops/deployment/docker.md
+++ b/docs/ops/deployment/docker.md
@@ -46,12 +46,12 @@ For example, you can use the following aliases:
* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-<span class="label label-info">Note</span> Prio to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
that do not include a bundled Hadoop distribution.
-## How to run Flink image
+## How to run a Flink image
The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
You can run its entry point in the following modes:
@@ -70,21 +70,43 @@ and deploys *TaskManagers* on demand so that you do not have to do it manually.
The next chapters describe how to start a single Flink Docker container for various purposes.
+Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+
+We recommend using [Docker Compose](docker.html#session-cluster-with-docker-compose) or [Docker Swarm](docker.html#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
+
### Start a Session Cluster
A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
-To deploy a *Flink Session cluster* with Docker, you need to start a *Flink Master* container:
+To deploy a *Flink Session cluster* with Docker, you need to start a *Flink Master* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
+```sh
+FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+docker network create flink-network
+```
+
+Then we launch the JobManager:
```sh
-docker run flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} jobmanager
+docker run \
+ --rm \
+ --name=jobmanager \
+ --network flink-network \
+ -p 8081:8081 \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} jobmanager
```
and one or more *TaskManager* containers:
```sh
-docker run flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
+docker run \
+ --rm \
+ --name=taskmanager \
+ --network flink-network \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+
### Start a Job Cluster
A *Flink Job cluster* is a dedicated cluster which runs a single job.
@@ -105,18 +127,26 @@ To make the **job artifacts available** locally in the container, you can
the *Flink Master* and *TaskManagers*:
```sh
+ FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+ docker network create flink-network
+
docker run \
--mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
--mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
+ --rm \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ --name=jobmanager \
+ --network flink-network \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} standalone-job \
--job-classname com.job.ClassName \
- --job-id <job id> \
+ [--job-id <job id>] \
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
[job arguments]
docker run \
--mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
--mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
@@ -133,7 +163,7 @@ the *Flink Master* and *TaskManagers*:
docker run \
flink_with_job_artifacts standalone-job \
--job-classname com.job.ClassName \
- --job-id <job id> \
+ [--job-id <job id>] \
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
[job arguments]
@@ -186,7 +216,7 @@ docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable
The [`jobmanager.rpc.address`](../config.html#jobmanager-rpc-address) option must be configured, others are optional to set.
The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
-the same way as in the `flink-conf.yaml`.
+the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
### Provide custom configuration
@@ -215,17 +245,16 @@ The `flink-conf.yaml` file must have write permission so that the Docker entry p
### Using plugins
As described in the [plugins]({{ site.baseurl }}/ops/plugins.html) documentation page: in order to use plugins they must be
-copied to the correct location in the Flink installation for them to work.
+copied to the correct location in the Flink installation in the Docker container for them to work.
-If you want to enable plugins provided with Flink, you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS`
-when you run the Flink image.
-The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`.
+If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
+The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
- ```sh
+```sh
docker run \
--env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
- ```
+```
There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
@@ -367,6 +396,7 @@ The next chapters show examples of configuration files to run Flink.
### Session Cluster with Docker Compose
+**docker-compose.yml:**
```yaml
version: "2.2"
services:
@@ -399,6 +429,7 @@ The artifacts must be available in the Flink containers, check details [here](#s
See also [how to specify the Flink Master arguments](#flink-master-additional-command-line-arguments)
in the `command` for the `jobmanager` service.
+**docker-compose.yml:**
```yaml
version: "2.2"
services:
diff --git a/docs/ops/deployment/docker.zh.md b/docs/ops/deployment/docker.zh.md
index 1cab634..6a4aea7 100644
--- a/docs/ops/deployment/docker.zh.md
+++ b/docs/ops/deployment/docker.zh.md
@@ -46,12 +46,12 @@ For example, you can use the following aliases:
* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`
-<span class="label label-info">Note</span> Prio to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
+<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
that do not include a bundled Hadoop distribution.
-## How to run Flink image
+## How to run a Flink image
The Flink image contains a regular Flink distribution with its default configuration and a standard entry point script.
You can run its entry point in the following modes:
@@ -70,21 +70,43 @@ and deploys *TaskManagers* on demand so that you do not have to do it manually.
The next chapters describe how to start a single Flink Docker container for various purposes.
+Once you've started Flink on Docker, you can access the Flink Webfrontend on [localhost:8081](http://localhost:8081/#/overview) or submit jobs like this `./bin/flink run ./examples/streaming/TopSpeedWindowing.jar`.
+
+We recommend using [Docker Compose](docker.html#session-cluster-with-docker-compose) or [Docker Swarm](docker.html#session-cluster-with-docker-swarm) for deploying Flink as a Session Cluster to ease system configuration.
+
### Start a Session Cluster
A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed.
-To deploy a *Flink Session cluster* with Docker, you need to start a *Flink Master* container:
+To deploy a *Flink Session cluster* with Docker, you need to start a *Flink Master* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
+```sh
+FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+docker network create flink-network
+```
+
+Then we launch the JobManager:
```sh
-docker run flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} jobmanager
+docker run \
+ --rm \
+ --name=jobmanager \
+ --network flink-network \
+ -p 8081:8081 \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} jobmanager
```
and one or more *TaskManager* containers:
```sh
-docker run flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
+docker run \
+ --rm \
+ --name=taskmanager \
+ --network flink-network \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
+
### Start a Job Cluster
A *Flink Job cluster* is a dedicated cluster which runs a single job.
@@ -105,18 +127,26 @@ To make the **job artifacts available** locally in the container, you can
the *Flink Master* and *TaskManagers*:
```sh
+ FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+ docker network create flink-network
+
docker run \
--mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
--mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
+ --rm \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ --name=jobmanager \
+ --network flink-network \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} standalone-job \
--job-classname com.job.ClassName \
- --job-id <job id> \
+ [--job-id <job id>] \
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
[job arguments]
docker run \
--mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
--mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} taskmanager
```
@@ -133,7 +163,7 @@ the *Flink Master* and *TaskManagers*:
docker run \
flink_with_job_artifacts standalone-job \
--job-classname com.job.ClassName \
- --job-id <job id> \
+ [--job-id <job id>] \
[--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
[job arguments]
@@ -186,7 +216,7 @@ docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{% if site.is_stable
The [`jobmanager.rpc.address`](../config.html#jobmanager-rpc-address) option must be configured, others are optional to set.
The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line,
-the same way as in the `flink-conf.yaml`.
+the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`.
### Provide custom configuration
@@ -214,18 +244,17 @@ The `flink-conf.yaml` file must have write permission so that the Docker entry p
### Using plugins
-As described in the [plugins]({{ site.baseurl }}/zh/ops/plugins.html) documentation page: in order to use plugins they must be
-copied to the correct location in the Flink installation for them to work.
+As described in the [plugins]({{ site.baseurl }}/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation in the Docker container for them to work.
-If you want to enable plugins provided with Flink, you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS`
-when you run the Flink image.
-The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`.
+If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image.
+The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{site.version}}.jar`
- ```sh
+```sh
docker run \
--env ENABLE_BUILT_IN_PLUGINS=flink-plugin1.jar;flink-plugin2.jar \
flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %} <jobmanager|standalone-job|taskmanager>
- ```
+```
There are also more [advanced ways](#advanced-customization) for customizing the Flink image.
@@ -367,6 +396,7 @@ The next chapters show examples of configuration files to run Flink.
### Session Cluster with Docker Compose
+**docker-compose.yml:**
```yaml
version: "2.2"
services:
@@ -399,6 +429,7 @@ The artifacts must be available in the Flink containers, check details [here](#s
See also [how to specify the Flink Master arguments](#flink-master-additional-command-line-arguments)
in the `command` for the `jobmanager` service.
+**docker-compose.yml:**
```yaml
version: "2.2"
services: