You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/08/19 05:10:39 UTC
[GitHub] [flink-web] rmetzger opened a new pull request #370: Add a blog post about the current state of Flink on Docker
rmetzger opened a new pull request #370:
URL: https://github.com/apache/flink-web/pull/370
This blog post hasn't been proofread by anybody beforehand, so this is mostly for collecting early feedback.
I'll cleanup the commit history and date in the blog post later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] asfgit closed pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #370:
URL: https://github.com/apache/flink-web/pull/370
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] azagrebin commented on a change in pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #370:
URL: https://github.com/apache/flink-web/pull/370#discussion_r473789842
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
+
+Let's quickly break down the recent improvements:
+
+- Reduce confusion: Flink used to have 2 Dockerfiles and a 3rd file maintained outside of the official repository — all with different features and varying stability. Now, we have one central place for all images: [apache/flink-docker](https://github.com/apache/flink-docker).
+
+ Here, we keep all the Dockerfiles for the different releases. Check out the [detailed readme](https://github.com/apache/flink-docker/blob/master/README.md) of that repository for further explanation on the different branches, as well as the [Flink Improvement Proposal (FLIP-111)](https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification) that contains the detailed planning.
+
+ The apache/flink-docker repository also seeds the o[fficial Flink image on Docker Hub](https://hub.docker.com/_/flink).
+
+- Improve Usability: The Dockerfiles are used for various purposes: [Native docker deployments](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/docker.html), [Flink on Kubernetes](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html), the (unofficial) [Flink helm example](https://github.com/docker-flink/examples) and the project's [internal end to end tests](https://github.com/apache/flink/tree/master/flink-end-to-end-tests). With one unified image, all these consumers of the images benefit from the same set of features, documentation and testing.
+
+ The new images support passing configuration variables via a `FLINK_PROPERTIES` environment variable. Users can enable default plugins with the `ENABLE_BUILT_IN_PLUGINS`environment variable. The images also allow loading custom jar paths and configuration files.
+
+Looking into the future, there are already some interesting potential improvements lined up:
+
+- [Java 11 Docker images](https://issues.apache.org/jira/browse/FLINK-16260) (already completed)
+- [Use vanilla docker-entrypoint with flink-kubernetes](https://issues.apache.org/jira/browse/FLINK-15793) (in progress)
+- [History server support](https://issues.apache.org/jira/browse/FLINK-17167)
+- [Support for OpenShift](https://issues.apache.org/jira/browse/FLINK-15587)
+
+## How do I get started?
+
+This is a short tutorial on how to start a Flink Session Cluster with docker.
Review comment:
again I would either make `a Flink Session Cluster with docker` clickable into the docs or add `see also details in docs link`.
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
+
+Let's quickly break down the recent improvements:
+
+- Reduce confusion: Flink used to have 2 Dockerfiles and a 3rd file maintained outside of the official repository — all with different features and varying stability. Now, we have one central place for all images: [apache/flink-docker](https://github.com/apache/flink-docker).
+
+ Here, we keep all the Dockerfiles for the different releases. Check out the [detailed readme](https://github.com/apache/flink-docker/blob/master/README.md) of that repository for further explanation on the different branches, as well as the [Flink Improvement Proposal (FLIP-111)](https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification) that contains the detailed planning.
+
+ The apache/flink-docker repository also seeds the o[fficial Flink image on Docker Hub](https://hub.docker.com/_/flink).
Review comment:
```suggestion
The `apache/flink-docker` repository also seeds the [official Flink image on Docker Hub](https://hub.docker.com/_/flink).
```
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
+
+Let's quickly break down the recent improvements:
+
+- Reduce confusion: Flink used to have 2 Dockerfiles and a 3rd file maintained outside of the official repository — all with different features and varying stability. Now, we have one central place for all images: [apache/flink-docker](https://github.com/apache/flink-docker).
+
+ Here, we keep all the Dockerfiles for the different releases. Check out the [detailed readme](https://github.com/apache/flink-docker/blob/master/README.md) of that repository for further explanation on the different branches, as well as the [Flink Improvement Proposal (FLIP-111)](https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification) that contains the detailed planning.
+
+ The apache/flink-docker repository also seeds the o[fficial Flink image on Docker Hub](https://hub.docker.com/_/flink).
+
+- Improve Usability: The Dockerfiles are used for various purposes: [Native docker deployments](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/docker.html), [Flink on Kubernetes](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html), the (unofficial) [Flink helm example](https://github.com/docker-flink/examples) and the project's [internal end to end tests](https://github.com/apache/flink/tree/master/flink-end-to-end-tests). With one unified image, all these consumers of the images benefit from the same set of features, documentation and testing.
+
+ The new images support passing configuration variables via a `FLINK_PROPERTIES` environment variable. Users can enable default plugins with the `ENABLE_BUILT_IN_PLUGINS`environment variable. The images also allow loading custom jar paths and configuration files.
+
+Looking into the future, there are already some interesting potential improvements lined up:
+
+- [Java 11 Docker images](https://issues.apache.org/jira/browse/FLINK-16260) (already completed)
+- [Use vanilla docker-entrypoint with flink-kubernetes](https://issues.apache.org/jira/browse/FLINK-15793) (in progress)
+- [History server support](https://issues.apache.org/jira/browse/FLINK-17167)
+- [Support for OpenShift](https://issues.apache.org/jira/browse/FLINK-15587)
+
+## How do I get started?
+
+This is a short tutorial on how to start a Flink Session Cluster with docker.
+
+A *Flink Session cluster* can be used to run multiple jobs. Each job needs to be submitted to the cluster after it has been deployed. To deploy a *Flink Session cluster* with Docker, you need to start a *JobManager* container. To enable communication between the containers, we first set a required Flink configuration property and create a network:
+
+```
+FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
+docker network create flink-network
+```
+
+Then we launch the JobManager:
+
+```
+docker run \
+ --rm \
+ --name=jobmanager \
+ --network flink-network \
+ -p 8081:8081 \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ flink:1.11.1 jobmanager
+```
+and one or more *TaskManager* containers:
+
+```
+docker run \
+ --rm \
+ --name=taskmanager \
+ --network flink-network \
+ --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
+ flink:1.11.1 taskmanager
+```
+
+You now have a fully functional Flink cluster running! You can access the the web front end here: [localhost:8081](http://localhost:8081/).
+
+Let's now submit one of Flink's example jobs:
+
+```bash
+# 1: (optional) Download the Flink distribution, and unpack it
+wget https://archive.apache.org/dist/flink/flink-1.11.1/flink-1.11.1-bin-scala_2.12.tgz
+tar xf flink-1.11.1-bin-scala_2.12.tgz
+cd flink-1.11.1
+
+# 2: Start the Flink job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+```
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/flink-docker/flink-docker.gif" width="882px" height="730px" alt="Demo video"/>
+</center>
+
Review comment:
I would also briefly mention that one could also use docs link to deploy the job mode, just to denote that it is also possible and we envisioned it.
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
+
+Let's quickly break down the recent improvements:
+
+- Reduce confusion: Flink used to have 2 Dockerfiles and a 3rd file maintained outside of the official repository — all with different features and varying stability. Now, we have one central place for all images: [apache/flink-docker](https://github.com/apache/flink-docker).
+
+ Here, we keep all the Dockerfiles for the different releases. Check out the [detailed readme](https://github.com/apache/flink-docker/blob/master/README.md) of that repository for further explanation on the different branches, as well as the [Flink Improvement Proposal (FLIP-111)](https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification) that contains the detailed planning.
+
+ The apache/flink-docker repository also seeds the o[fficial Flink image on Docker Hub](https://hub.docker.com/_/flink).
Review comment:
or some other highlighting for `apache/flink-docker`
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
Review comment:
nit:
I am not a native speaker.
`upgrading experience` sounds somewhat strange for me,
maybe because I've encountered only `improving experience` so far
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
+
+Let's quickly break down the recent improvements:
+
+- Reduce confusion: Flink used to have 2 Dockerfiles and a 3rd file maintained outside of the official repository — all with different features and varying stability. Now, we have one central place for all images: [apache/flink-docker](https://github.com/apache/flink-docker).
+
+ Here, we keep all the Dockerfiles for the different releases. Check out the [detailed readme](https://github.com/apache/flink-docker/blob/master/README.md) of that repository for further explanation on the different branches, as well as the [Flink Improvement Proposal (FLIP-111)](https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification) that contains the detailed planning.
+
+ The apache/flink-docker repository also seeds the o[fficial Flink image on Docker Hub](https://hub.docker.com/_/flink).
+
+- Improve Usability: The Dockerfiles are used for various purposes: [Native docker deployments](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/docker.html), [Flink on Kubernetes](https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html), the (unofficial) [Flink helm example](https://github.com/docker-flink/examples) and the project's [internal end to end tests](https://github.com/apache/flink/tree/master/flink-end-to-end-tests). With one unified image, all these consumers of the images benefit from the same set of features, documentation and testing.
+
+ The new images support passing configuration variables via a `FLINK_PROPERTIES` environment variable. Users can enable default plugins with the `ENABLE_BUILT_IN_PLUGINS`environment variable. The images also allow loading custom jar paths and configuration files.
Review comment:
It may be helpful to turn the phrases for key features into links to the corresponding doc chapters (e.g. FLINK_PROPERTIES, ENABLE_BUILT_IN_PLUGINS, custom jar paths and configuration files). I know we already have a general link but I believe more links do not hurt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #370:
URL: https://github.com/apache/flink-web/pull/370#discussion_r473867137
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
Review comment:
okay, I'll use "improve"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] sjwiesman commented on a change in pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #370:
URL: https://github.com/apache/flink-web/pull/370#discussion_r473175127
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
Review comment:
quick drive-by suggestion
```suggestion
With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
The Flink community recently put some effort into upgrading the Docker experience for our users with the goal to reduce confusion and improve usability.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #370:
URL: https://github.com/apache/flink-web/pull/370#discussion_r473850331
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
Review comment:
... elevating the Docker experience to a new level ... ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #370:
URL: https://github.com/apache/flink-web/pull/370#discussion_r473846656
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
Review comment:
Damn, me trying to sound eloquent often fires back :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] rmetzger commented on pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #370:
URL: https://github.com/apache/flink-web/pull/370#issuecomment-677515170
Thanks a lot for your reviews! I pushed a new version. Unless there's any more feedback, I'll merge it later today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #370: Add a blog post about the current state of Flink on Docker
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #370:
URL: https://github.com/apache/flink-web/pull/370#discussion_r473849162
##########
File path: _posts/2020-08-20-flink-docker.md
##########
@@ -0,0 +1,90 @@
+---
+layout: post
+title: "The State of Flink on Docker"
+date: 2020-08-08T00:00:00.000Z
+authors:
+- rmetzger:
+ name: "Robert Metzger"
+ twitter: rmetzger_
+categories: news
+
+excerpt: This blog post gives an update on the recent developments of Flink's support for Docker.
+---
+
+The Flink community recently put some effort into upgrading the Docker experience for our users. The goal was to reduce confusion and improve usability. With over 50 million downloads from Docker Hub, the Flink docker images are a very popular deployment option.
Review comment:
Seth didn't correct the word in his review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org