You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tr...@apache.org on 2020/03/19 15:02:59 UTC
[flink] branch master updated (b9943dc -> b237b00)
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from b9943dc [FLINK-16676][python][test] Fix ValidationPipelineTest.test_pipeline_from_invalid_json failure in Azure
new 75b240d [hotfix][docs] Fix broken links
new b237b00 [FLINK-16293] Document using plugins in Docker and Kubernetes
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/ops/deployment/docker.md | 31 +++++++++++++++++++++++++++++
docs/ops/deployment/docker.zh.md | 31 +++++++++++++++++++++++++++++
docs/ops/deployment/kubernetes.md | 25 +++++++++++++++++------
docs/ops/deployment/kubernetes.zh.md | 13 ++++++++++++
docs/ops/deployment/native_kubernetes.md | 26 ++++++++++++++++++++++++
docs/ops/deployment/native_kubernetes.zh.md | 26 ++++++++++++++++++++++++
docs/ops/plugins.md | 2 +-
docs/ops/plugins.zh.md | 2 +-
8 files changed, 148 insertions(+), 8 deletions(-)
[flink] 01/02: [hotfix][docs] Fix broken links
Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 75b240d49c0944818a3de37af07040432958ad5b
Author: Niels Basjes <ni...@basjes.nl>
AuthorDate: Wed Feb 26 18:27:13 2020 +0100
[hotfix][docs] Fix broken links
---
docs/ops/plugins.md | 2 +-
docs/ops/plugins.zh.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/ops/plugins.md b/docs/ops/plugins.md
index 759d5dc..ea203ff 100644
--- a/docs/ops/plugins.md
+++ b/docs/ops/plugins.md
@@ -70,7 +70,7 @@ possible across Flink core, plugins, and user code.
## File Systems
-All [file systems](filesystems/index) **except MapR** are pluggable. That means they can and should
+All [file systems](filesystems) **except MapR** are pluggable. That means they can and should
be used as plugins. To use a pluggable file system, copy the corresponding JAR file from the `opt`
directory to a directory under `plugins` directory of your Flink distribution before starting Flink,
e.g.
diff --git a/docs/ops/plugins.zh.md b/docs/ops/plugins.zh.md
index 759d5dc..ea203ff 100644
--- a/docs/ops/plugins.zh.md
+++ b/docs/ops/plugins.zh.md
@@ -70,7 +70,7 @@ possible across Flink core, plugins, and user code.
## File Systems
-All [file systems](filesystems/index) **except MapR** are pluggable. That means they can and should
+All [file systems](filesystems) **except MapR** are pluggable. That means they can and should
be used as plugins. To use a pluggable file system, copy the corresponding JAR file from the `opt`
directory to a directory under `plugins` directory of your Flink distribution before starting Flink,
e.g.
[flink] 02/02: [FLINK-16293] Document using plugins in Docker and
Kubernetes
Posted by tr...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit b237b0081473016aff558f6c1393357d9409d1e3
Author: Niels Basjes <ni...@basjes.nl>
AuthorDate: Wed Mar 4 10:29:16 2020 +0100
[FLINK-16293] Document using plugins in Docker and Kubernetes
This closes #11228.
---
docs/ops/deployment/docker.md | 31 +++++++++++++++++++++++++++++
docs/ops/deployment/docker.zh.md | 31 +++++++++++++++++++++++++++++
docs/ops/deployment/kubernetes.md | 25 +++++++++++++++++------
docs/ops/deployment/kubernetes.zh.md | 13 ++++++++++++
docs/ops/deployment/native_kubernetes.md | 26 ++++++++++++++++++++++++
docs/ops/deployment/native_kubernetes.zh.md | 26 ++++++++++++++++++++++++
6 files changed, 146 insertions(+), 6 deletions(-)
diff --git a/docs/ops/deployment/docker.md b/docs/ops/deployment/docker.md
index 0c86074..a716206 100644
--- a/docs/ops/deployment/docker.md
+++ b/docs/ops/deployment/docker.md
@@ -68,6 +68,37 @@ Therefore, one needs to build a dedicated container image for every job.
The `flink-container` module contains a `build.sh` script which can be used to create such an image.
Please see the [instructions](https://github.com/apache/flink/blob/{{ site.github_branch }}/flink-container/docker/README.md) for more details.
+## Using plugins
+As described in the [plugins]({{ site.baseurl }}/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation for them to work.
+
+When running Flink from one of the provided Docker images by default no plugins have been activated.
+The simplest way to enable plugins is to modify the provided official Flink docker images by adding
+an additional layer. This does however assume you have a docker registry available where you can push images to and
+that is accessible by your cluster.
+
+As an example assume you want to enable the [S3]({{ site.baseurl }}/ops/filesystems/s3.html) plugins in your installation.
+
+Create a Dockerfile with a content something like this: {% highlight dockerfile %}
+# On which specific version of Flink is this based?
+# Check https://hub.docker.com/_/flink?tab=tags for current options
+FROM flink:{{ site.version }}-scala_2.12
+
+# Install Flink S3 FS Presto plugin
+RUN mkdir /opt/flink/plugins/s3-fs-presto && cp /opt/flink/opt/flink-s3-fs-presto* /opt/flink/plugins/s3-fs-presto
+
+# Install Flink S3 FS Hadoop plugin
+RUN mkdir /opt/flink/plugins/s3-fs-hadoop && cp /opt/flink/opt/flink-s3-fs-hadoop* /opt/flink/plugins/s3-fs-hadoop
+{% endhighlight %}
+
+Then build and push that image to your registry
+{% highlight bash %}
+docker build -t docker.example.nl/flink:{{ site.version }}-scala_2.12-s3 .
+docker push docker.example.nl/flink:{{ site.version }}-scala_2.12-s3
+{% endhighlight %}
+
+Now you can reference this image in your cluster deployment and the installed plugins are available for use.
+
## Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a convenient way to run a
diff --git a/docs/ops/deployment/docker.zh.md b/docs/ops/deployment/docker.zh.md
index d82d736..69ac45e 100644
--- a/docs/ops/deployment/docker.zh.md
+++ b/docs/ops/deployment/docker.zh.md
@@ -68,6 +68,37 @@ Therefore, one needs to build a dedicated container image for every job.
The `flink-container` module contains a `build.sh` script which can be used to create such an image.
Please see the [instructions](https://github.com/apache/flink/blob/{{ site.github_branch }}/flink-container/docker/README.md) for more details.
+## Using plugins
+As described in the [plugins]({{ site.baseurl }}/zh/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the Flink installation for them to work.
+
+When running Flink from one of the provided Docker images by default no plugins have been activated.
+The simplest way to enable plugins is to modify the provided official Flink docker images by adding
+an additional layer. This does however assume you have a docker registry available where you can push images to and
+that is accessible by your cluster.
+
+As an example assume you want to enable the [S3]({{ site.baseurl }}/zh/ops/filesystems/s3.html) plugins in your installation.
+
+Create a Dockerfile with a content something like this: {% highlight dockerfile %}
+# On which specific version of Flink is this based?
+# Check https://hub.docker.com/_/flink?tab=tags for current options
+FROM flink:{{ site.version }}-scala_2.12
+
+# Install Flink S3 FS Presto plugin
+RUN mkdir /opt/flink/plugins/s3-fs-presto && cp /opt/flink/opt/flink-s3-fs-presto* /opt/flink/plugins/s3-fs-presto
+
+# Install Flink S3 FS Hadoop plugin
+RUN mkdir /opt/flink/plugins/s3-fs-hadoop && cp /opt/flink/opt/flink-s3-fs-hadoop* /opt/flink/plugins/s3-fs-hadoop
+{% endhighlight %}
+
+Then build and push that image to your registry
+{% highlight bash %}
+docker build -t docker.example.nl/flink:{{ site.version }}-scala_2.12-s3 .
+docker push docker.example.nl/flink:{{ site.version }}-scala_2.12-s3
+{% endhighlight %}
+
+Now you can reference this image in your cluster deployment and then these plugins are available for use.
+
## Flink with Docker Compose
[Docker Compose](https://docs.docker.com/compose/) is a convenient way to run a
diff --git a/docs/ops/deployment/kubernetes.md b/docs/ops/deployment/kubernetes.md
index f30f21a..d798bff 100644
--- a/docs/ops/deployment/kubernetes.md
+++ b/docs/ops/deployment/kubernetes.md
@@ -36,13 +36,13 @@ Please follow [Kubernetes' setup guide](https://kubernetes.io/docs/setup/) in or
If you want to run Kubernetes locally, we recommend using [MiniKube](https://kubernetes.io/docs/setup/minikube/).
<div class="alert alert-info" markdown="span">
- <strong>Note:</strong> If using MiniKube please make sure to execute `minikube ssh 'sudo ip link set docker0 promisc on'` before deploying a Flink cluster.
- Otherwise Flink components are not able to self reference themselves through a Kubernetes service.
+ <strong>Note:</strong> If using MiniKube please make sure to execute `minikube ssh 'sudo ip link set docker0 promisc on'` before deploying a Flink cluster.
+ Otherwise Flink components are not able to self reference themselves through a Kubernetes service.
</div>
## Flink session cluster on Kubernetes
-A Flink session cluster is executed as a long-running Kubernetes Deployment.
+A Flink session cluster is executed as a long-running Kubernetes Deployment.
Note that you can run multiple Flink jobs on a session cluster.
Each job needs to be submitted to the cluster after the cluster has been deployed.
@@ -91,15 +91,28 @@ In order to terminate the Flink session cluster, use `kubectl`:
## Flink job cluster on Kubernetes
-A Flink job cluster is a dedicated cluster which runs a single job.
-The job is part of the image and, thus, there is no extra job submission needed.
+A Flink job cluster is a dedicated cluster which runs a single job.
+The job is part of the image and, thus, there is no extra job submission needed.
### Creating the job-specific image
The Flink job cluster image needs to contain the user code jars of the job for which the cluster is started.
Therefore, one needs to build a dedicated container image for every job.
Please follow these [instructions](https://github.com/apache/flink/blob/{{ site.github_branch }}/flink-container/docker/README.md) to build the Docker image.
-
+
+### Using plugins
+
+As described in the [plugins]({{ site.baseurl }}/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the flink installation for them to work.
+
+The simplest way to enable plugins for use on Kubernetes is to modify the provided Flink docker image by adding
+an additional layer. This does however assume you have a docker registry available where you can push images to and
+that is accessible by your Kubernetes cluster.
+
+How this can be done is described on the [Docker Setup]({{ site.baseurl }}/zh/ops/deployment/docker.html#using-plugins) page.
+
+With such an image created you can now start your Kubernetes based Flink cluster which can use the enabled plugins.
+
### Deploy Flink job cluster on Kubernetes
In order to deploy the a job cluster on Kubernetes please follow these [instructions](https://github.com/apache/flink/blob/{{ site.github_branch }}/flink-container/kubernetes/README.md#deploy-flink-job-cluster).
diff --git a/docs/ops/deployment/kubernetes.zh.md b/docs/ops/deployment/kubernetes.zh.md
index 014c937..f5dd31a 100644
--- a/docs/ops/deployment/kubernetes.zh.md
+++ b/docs/ops/deployment/kubernetes.zh.md
@@ -100,6 +100,19 @@ The Flink job cluster image needs to contain the user code jars of the job for w
Therefore, one needs to build a dedicated container image for every job.
Please follow these [instructions](https://github.com/apache/flink/blob/{{ site.github_branch }}/flink-container/docker/README.md) to build the Docker image.
+### Using plugins
+
+As described in the [plugins]({{ site.baseurl }}/zh/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the flink installation for them to work.
+
+The simplest way to enable plugins for use on Kubernetes is to modify the provided Flink docker image by adding
+an additional layer. This does however assume you have a docker registry available where you can push images to and
+that is accessible by your Kubernetes cluster.
+
+How this can be done is described on the [Docker Setup]({{ site.baseurl }}/zh/ops/deployment/docker.html#using-plugins) page.
+
+With such an image created you can now start your Kubernetes based Flink cluster which can use the enabled plugins.
+
### Deploy Flink job cluster on Kubernetes
In order to deploy the a job cluster on Kubernetes please follow these [instructions](https://github.com/apache/flink/blob/{{ site.github_branch }}/flink-container/kubernetes/README.md#deploy-flink-job-cluster).
diff --git a/docs/ops/deployment/native_kubernetes.md b/docs/ops/deployment/native_kubernetes.md
index d543609..b380611 100644
--- a/docs/ops/deployment/native_kubernetes.md
+++ b/docs/ops/deployment/native_kubernetes.md
@@ -154,6 +154,32 @@ appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m
If the pod is running, you can use `kubectl exec -it <PodName> bash` to tunnel in and view the logs or debug the process.
+## Using plugins
+
+As described in the [plugins]({{ site.baseurl }}/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the flink installation for them to work.
+
+The simplest way to enable plugins for use on Kubernetes is to modify the provided official Flink docker images by adding
+an additional layer. This does however assume you have a docker registry available where you can push images to and
+that is accessible by your Kubernetes cluster.
+
+How this can be done is described on the [Docker Setup]({{ site.baseurl }}/ops/deployment/docker.html#using-plugins) page.
+
+With such an image created you can now start your Kubernetes based Flink session cluster with the additional parameter
+`kubernetes.container.image` which must specify the image that was created: `docker.example.nl/flink:{{ site.version }}-2.12-s3`
+
+Extending the above example command to start the session cluster makes it this:
+
+{% highlight bash %}
+./bin/kubernetes-session.sh \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dresourcemanager.taskmanager-timeout=3600000 \
+ -Dkubernetes.container.image=docker.example.nl/flink:{{ site.version }}-2.12-s3
+{% endhighlight %}
+
## Kubernetes concepts
### Namespaces
diff --git a/docs/ops/deployment/native_kubernetes.zh.md b/docs/ops/deployment/native_kubernetes.zh.md
index 1337dc0..b9c328d 100644
--- a/docs/ops/deployment/native_kubernetes.zh.md
+++ b/docs/ops/deployment/native_kubernetes.zh.md
@@ -154,6 +154,32 @@ appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m
If the pod is running, you can use `kubectl exec -it <PodName> bash` to tunnel in and view the logs or debug the process.
+## Using plugins
+
+As described in the [plugins]({{ site.baseurl }}/zh/ops/plugins.html) documentation page: in order to use plugins they must be
+copied to the correct location in the flink installation for them to work.
+
+The simplest way to enable plugins for use on Kubernetes is to modify the provided official Flink docker images by adding
+an additional layer. This does however assume you have a docker registry available where you can push images to and
+that is accessible by your Kubernetes cluster.
+
+How this can be done is described on the [Docker Setup]({{ site.baseurl }}/zh/ops/deployment/docker.html#using-plugins) page.
+
+With such an image created you can now start your Kubernetes based Flink session cluster with the additional parameter
+`kubernetes.container.image` which must specify the image that was created: `docker.example.nl/flink:{{ site.version }}-2.12-s3`
+
+Extending the above example command to start the session cluster makes it this:
+
+{% highlight bash %}
+./bin/kubernetes-session.sh \
+ -Dkubernetes.cluster-id=<ClusterId> \
+ -Dtaskmanager.memory.process.size=4096m \
+ -Dkubernetes.taskmanager.cpu=2 \
+ -Dtaskmanager.numberOfTaskSlots=4 \
+ -Dresourcemanager.taskmanager-timeout=3600000 \
+ -Dkubernetes.container.image=docker.example.nl/flink:{{ site.version }}-2.12-s3
+{% endhighlight %}
+
## Kubernetes concepts
### Namespaces