You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by iemejia <gi...@git.apache.org> on 2016/06/28 13:05:41 UTC
[GitHub] flink pull request #2176: Flink 4118
GitHub user iemejia opened a pull request:
https://github.com/apache/flink/pull/2176
Flink 4118
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/iemejia/flink FLINK-4118
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2176
----
commit 7f7f24a1c31aa3e9b011fb492280389ac500fff4
Author: Isma�l Mej�a <ie...@gmail.com>
Date: 2016-06-23T16:39:50Z
[FLINK-4118] Update docker image to 1.0.3 and remove unneeded deps
Some of the changes include:
- Remove unneeded dependencies (nano, wget)
- Remove apt lists to reduce image size
- Reduce number of layers on the docker image (best docker practice)
- Remove useless variables and base the code in generic ones e.g.
FLINK_HOME
- Change the default JDK from oracle to openjdk-8-jre-headless, based on
two reasons:
1. You cannot legally repackage the oracle jdk in docker images
2. The open-jdk headless is more appropriate for a server image (no GUI stuff)
- Return port assignation to the standard FLINK one:
Variable: docker-flink -> flink
taskmanager.rpc.port: 6121 -> 6122
taskmanager.data.port: 6122 -> 6121
jobmanager.web.port: 8080 -> 8081
commit 04c18edbbb6b109c1b23c28c17f82bde080b8686
Author: Isma�l Mej�a <ie...@gmail.com>
Date: 2016-06-24T14:52:22Z
[FLINK-4118] Base the image on the official java alpine and remove ssh
commit eff306e80ffe605f6f94d2b5764520fe6462b1f7
Author: Isma�l Mej�a <ie...@gmail.com>
Date: 2016-06-27T11:23:10Z
[FLINK-4118] Remove unused configuration files and fix README
commit e386b5e99c086e3279454a3e574edc7a02c54cdb
Author: Isma�l Mej�a <ie...@gmail.com>
Date: 2016-06-27T15:01:13Z
[FLINK-4118] Update compose to v2 + add new entrypoint for direct execution
commit 34f18849cd610199805bc396b2ac850242f5ab6d
Author: Isma�l Mej�a <ie...@gmail.com>
Date: 2016-06-27T16:52:48Z
[FLINK-4118] Change default entrypoint so users can run flink as a client too
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
One last thing I would like to try is running a job from an existing Flink installation using `$FLINK_HOME/bin/flink run -m <jobmanager:port> <your_jar>` as described in the README.
I suspect it has something to do with setting the right IP or network setting because Akka is very particular about the IP to which it is bound. Did you get this to work? I'm only managing to access the Web Dashboard.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
Re 1. I think this should be fine. There is some dynamic code generation but this uses Janino as a library so that shouldn't be a problem.
Re 2. I'm not aware of any but it should be easy to add them if we find any in the future.
Re 3. This is not easily possible since the `&` is hardcoded in `flink-daemon.sh`:
```
$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
```
I'd be very happy if you'd like to work on adding a flag or setting to make the daemons start non-daemonized. It should not be too hard to add that, IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69148656
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t "flink" flink
--- End diff --
On my machine this doesn't work, but this does: `docker build -t "flink" .` Is this maybe a leftover from the earlier version where there was a `flink` directory?
(Earlier I was using `sh build.sh`, that's why I didn't spot the problem.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69307034
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+ docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen session)
-`docker-compose up`
+ docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+ docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=<N>`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+ docker-compose scale taskmanager=<N>
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+ docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+ docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+ for i in $(docker ps --filter name=flink --format={{.ID}}); do
--- End diff --
Yep, the TaskManagers pull it from the JobManager which keeps it in a component called BlobManager.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
Nice work! I didn't know how to use docker but I managed to set it up and use the new version on OS X without a problem. So it seems to work well, and the code is a lot simpler and the image is smaller.
LGTM minus the one comment I had about the config file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69150030
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t "flink" flink
--- End diff --
Nice catch, just fixed :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69306386
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+ docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen session)
-`docker-compose up`
+ docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+ docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=<N>`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+ docker-compose scale taskmanager=<N>
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+ docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+ docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+ for i in $(docker ps --filter name=flink --format={{.ID}}); do
--- End diff --
Nice I didn't know that flink took care of this, fix in mins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69102159
--- Diff: flink-contrib/docker-flink/docker-entrypoint.sh ---
@@ -16,21 +18,23 @@
# limitations under the License.
################################################################################
-jobmanager.rpc.address: %jobmanager%
-jobmanager.rpc.port: 6123
-jobmanager.heap.mb: 128
-
-taskmanager.rpc.port: 6121
-taskmanager.data.port: 6122
-taskmanager.heap.mb: 256
-taskmanager.numberOfTaskSlots: %nb_slots%
-
-parallelization.degree.default: %parallelism%
+# general configuration
+sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: `grep -c ^processor /proc/cpuinfo`/g" $FLINK_HOME/conf/flink-conf.yaml
--- End diff --
This should only be executed on the JobManager since it will print `taskmanager_1 | sed: can't create temp file '/usr/local/flink/conf/flink-conf.yamlXXXXXX': Read-only file system` which might worry some users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
That's great to hear! I'll write something on the Beam ML thread.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69307221
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+ docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen session)
-`docker-compose up`
+ docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+ docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=<N>`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+ docker-compose scale taskmanager=<N>
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+ docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+ docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+ for i in $(docker ps --filter name=flink --format={{.ID}}); do
+ docker cp <your_jar> $i:/<your_path>
+ done
- Run a topology
-`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>`
+ docker run -it --rm flink:latest flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>
--- End diff --
I see. I can only speak from experience: I uploaded my jar to the JobManager container, then tried to run and it said that the jar was not available. I then found out that the run creates this new container that doesn't actually contain the jar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69306327
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+ docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen session)
-`docker-compose up`
+ docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+ docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=<N>`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+ docker-compose scale taskmanager=<N>
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+ docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+ docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+ for i in $(docker ps --filter name=flink --format={{.ID}}); do
+ docker cp <your_jar> $i:/<your_path>
+ done
- Run a topology
-`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>`
+ docker run -it --rm flink:latest flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>
--- End diff --
Yes, you are right, I hesitated about this, if you noticed, I added a third option apart of jobmanager/taskmanager to let the image open ended, my goal was in part that people could start an container with flink as a client, or for any other use. However this can be confusing (because people mostly expect to send ther local jar, but I will change it as you suggest in the README, and advanced users can do as they wish.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
I merged it, thanks again for your work!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2176
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
This should be ok now. In further commits I expect to fix the daemon thing + maybe add a HA version using zookeeper of the docker-compose file.
One more question aljoscha, I intend to add the Beam Flink runner and contribute a similar version into Beam, however I don't know what is the best approach for this, I just tried naively to put the jars in $FLINK_HOME/lib but it didn't work, any ideas ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Nice, I just fixed as you suggested. I have three questions:
1. This container is based on the Java JRE (to keep it small), Does Flink in any part do some magic that requires a full JDK (like live recompiles) ? If no, I think this is almost perfect now.
2. Are you aware of any other flink dependency that uses any native OS library (I ask this since this because if it is the case it must be added to the container, I did this for snappy because I found the issue while testing a Beam pipeline, but I don't know if there are others).
3. In the docker image I left supervisor because I didn't find an easy way to start flink in normal mode, the scripts to start both taskmanager and jobmanager go into daemon mode immediately, is there something that can be done to change this (this will reduce the image in 40 more MB), but well I can work on that for a different PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
I had two more comments about the README but after that it should be good to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Awasome, thanks @aljoscha, let's merge !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
Yes, this is exactly what I was trying on OS X. I'm quickly setting up a ubuntu VM to see if it works there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Great, thanks for your review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Sorry I had to rebase my previous PR but this is the definitive one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2176
You were right, I did exactly the same thing I did on OS X on a new Ubuntu 16.04 installation and it worked. \U0001f603
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: Flink 4118
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
The docker images script was simplified and the image size was reduced.
Previous image:
flink latest 6475add651c7 24 minutes ago 711.6 MB
Image after FLINK-4118
flink latest 555e60f24c10 20 seconds ago 252.5 MB
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69304962
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+ docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen session)
-`docker-compose up`
+ docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+ docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=<N>`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+ docker-compose scale taskmanager=<N>
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+ docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+ docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+ for i in $(docker ps --filter name=flink --format={{.ID}}); do
+ docker cp <your_jar> $i:/<your_path>
+ done
- Run a topology
-`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>`
+ docker run -it --rm flink:latest flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>
--- End diff --
AFAIK `docker run` starts a new container to run the command. This should probably be something like:
```
docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) flink run -m <jobmanager:port> -c <your_class> <your_jar> <your_params>
```
This would run the command on the existing JobManager container where the jar was previously uploaded.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2176: [FLINK-4118] The docker-flink image is outdated (1...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2176#discussion_r69304467
--- Diff: flink-contrib/docker-flink/README.md ---
@@ -1,80 +1,75 @@
-#Apache Flink cluster deployment on Docker using Docker-Compose
+Apache Flink cluster deployment on docker using docker-compose
-##Installation
-###Install Docker
+# Installation
+Install the most recent stable version of docker
https://docs.docker.com/installation/
-if you have issues with Docker-Compose versions incompatible with your version of Docker try
+Install the most recent stable version of docker-compose
+https://docs.docker.com/compose/install/
-`curl -sSL https://get.docker.com/ubuntu/ | sudo sh`
+# Build
-###Install Docker-Compose
+Images are based on the official Java Alpine (OpenJDK 8) image and run
+supervisord to stay alive when running containers. If you want to build the
+flink image run:
-```
-curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
+ sh build.sh
-chmod +x /usr/local/bin/docker-compose
-```
-
-###Get the repo
-
-###Build the images
+or
-Images are based on Ubuntu Trusty 14.04 and run Supervisord to stay alive when running containers.
+ docker build -t flink .
-The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.
+If you want to build the container for a specific version of flink/hadoop/scala
+you can configure it in the respective args:
-- Run `./build.sh`
+ docker build --build-arg FLINK_VERSION=1.0.3 --build-arg HADOOP_VERSION=26 --build-arg SCALA_VERSION=2.10 -t "flink:1.0.3-hadoop2.6-scala_2.10" flink
-###Deploy
+# Deploy
- Deploy cluster and see config/setup log output (best run in a screen session)
-`docker-compose up`
+ docker-compose up
- Deploy as a daemon (and return)
-`docker-compose up -d`
+ docker-compose up -d
- Scale the cluster up or down to *N* TaskManagers
-`docker-compose scale taskmanager=<N>`
-
-- Access the JobManager node with SSH (exposed on Port 220)
+ docker-compose scale taskmanager=<N>
-`ssh root@localhost -p 220`
+- Access the Job Manager container
-or on Mac OS X with boot2docker
-
-`ssh root@$(boot2docker ip) -p 220`
-
-The password is 'secret'
+ docker exec -it $(docker ps --filter name=flink_jobmanager --format={{.ID}}) /bin/sh
- Kill the cluster
-`docker-compose kill`
+ docker-compose kill
- Upload a jar to the cluster
-`scp -P 220 <your_jar> root@localhost:/<your_path>`
+ for i in $(docker ps --filter name=flink --format={{.ID}}); do
--- End diff --
The jar only needs to be uploaded to the JobManager container, so something like this should suffice:
```
docker cp <your_jar> $(docker ps --filter name=flink_jobmanager --format={{.ID}}):/<your_path>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2176: [FLINK-4118] The docker-flink image is outdated (1.0.2) a...
Posted by iemejia <gi...@git.apache.org>.
Github user iemejia commented on the issue:
https://github.com/apache/flink/pull/2176
Hi, I tested it with the basic word count and with the beam pipeline example that @ecesena put for his flink/beam demo.
I don't know if your case is maybe because you are running on docker for mac, but check two things.
1. docker assigns an internal IP to each running container (the one corresponding to the jobmanager is the one to put in the -m argument).
2. The path you pass in the examples must be accesible from the container, e.g. you must copy the file in all the nodes (as I mention in the README), or mount them with a volume (-v on docker).
e.g.
```
$FLINK_HOME/bin/flink run -m 172.18.0.2:6123 examples/batch/WordCount.jar --input file:///tmp/kinglear.txt
```
The kinglear file must be in all the nodes. I copy those like this:
```
for i in $(docker ps --filter name=flink --format={{.ID}}); do
docker cp ~/Desktop/flink-beam/kinglear.txt $i:/tmp/
done
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---