You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by jgrier <gi...@git.apache.org> on 2017/01/25 06:35:49 UTC
[GitHub] flink pull request #3205: [FLINK-5635] Improve Docker tooling to make it eas...
GitHub user jgrier opened a pull request:
https://github.com/apache/flink/pull/3205
[FLINK-5635] Improve Docker tooling to make it easier to build images and launch Flink via Docker tools
Improvements for Docker on Flink experience.
This PR depends on this other one: https://github.com/apache/flink/pull/3204
- Modifying Dockerfile to allow building from local flink-dist as well as release URLs
- Making image name configurable so it's easier for user's to use this to build an publish their own images
- Logging to stdout rather than just files
- Adding scripts to deploy seamlessly on Docker Swarm without host networking
- Updating Docker Compose scripts to work correctly for deploying locally
- Generally parameterizing things so these Docker scripts are more generally useful and self-documenting
- Provides options so that you can deploy multiple Flink services with unique names on Docker Swarm
- This should all work well with the new Docker "stack" stuff as well.
Example usage:
```
# Build a Docker image from your local flink build
./build.sh --from-local-dist --image-name "dataartisans/flink"
```
```
# Build a Docker image from an official release
./build.sh --from-release --flink-verison 1.2.0 --hadoop-version 2.7 --scala-version 2.11 --image-name "dataartisans/flink"
```
```
# Run with Docker Compose
docker-compose up -d
docker-compose scale taskmanager=4
```
```
# Run on Docker Swarm
./create-docker-swarm-service.sh --image-name "dataartisans/flink" my-flink-service my-flink-port
```
```
# Shut down
./remove-docker-swarm-service.sh --image-name "dataartisans/flink" my-flink-service
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jgrier/flink release-1.2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3205.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3205
----
commit d380e98b397c770ee4b7eb9e4abb2387e7672d4c
Author: Jamie Grier <ja...@gmail.com>
Date: 2017-01-21T00:12:17Z
Improvements for Docker on Flink experience.
Modifying Dockerfile to build from local flink-dist as well as release URLs.
Logging to stdout.
Adding scripts to deploy seamlessly on Docker Swarm.
Updating Docker Compose scripts to work correctly.
Parameterizing things so these Docker scripts are more generally useful.
commit 962290ab66d993cac4c4e233a017feb8a6eb1cc5
Author: Jamie Grier <ja...@gmail.com>
Date: 2017-01-21T00:14:27Z
Adding flag to flink-daemon.sh and related scripts so that we can actually log to the console -- which is better for Docker environments.
commit 66c2133ede0ed9914baba2fe7b93f1b9d585ede2
Author: Jamie Grier <ja...@gmail.com>
Date: 2017-01-25T05:45:23Z
Adding Apache license to new files.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3205: [FLINK-5635] Improve Docker tooling to make it eas...
Posted by ex00 <gi...@git.apache.org>.
Github user ex00 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3205#discussion_r100252102
--- Diff: flink-contrib/docker-flink/docker-compose.yml ---
@@ -16,21 +16,22 @@
# limitations under the License.
################################################################################
-version: "2"
+# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image name to use
+
+version: "2.1"
services:
jobmanager:
- image: flink
- container_name: "jobmanager"
+ image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6123"
--- End diff --
the variable in yml doesn't depends variable from Dockerfile.
see https://docs.docker.com/compose/compose-file/#variable-substitution
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3205: [FLINK-5635] Improve Docker tooling to make it eas...
Posted by tonycox <gi...@git.apache.org>.
Github user tonycox commented on a diff in the pull request:
https://github.com/apache/flink/pull/3205#discussion_r100159827
--- Diff: flink-contrib/docker-flink/docker-compose.yml ---
@@ -16,21 +16,22 @@
# limitations under the License.
################################################################################
-version: "2"
+# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image name to use
+
+version: "2.1"
services:
jobmanager:
- image: flink
- container_name: "jobmanager"
+ image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
expose:
- "6123"
--- End diff --
do we need this if it set in Dockerfile ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3205: [FLINK-5635] Improve Docker tooling to make it easier to ...
Posted by jgrier <gi...@git.apache.org>.
Github user jgrier commented on the issue:
https://github.com/apache/flink/pull/3205
@ex00 The `--from-local-dist` flag assumes you've already built Flink locally. If you run `mvn clean package -DskipTests` does this error persist?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3205: [FLINK-5635] Improve Docker tooling to make it eas...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3205
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3205: [FLINK-5635] Improve Docker tooling to make it eas...
Posted by tonycox <gi...@git.apache.org>.
Github user tonycox commented on a diff in the pull request:
https://github.com/apache/flink/pull/3205#discussion_r100156082
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,25 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory, a tarball on the local system, or a url to a tarball
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
mkdir -p $FLINK_INSTALL_PATH && \
--- End diff --
and what if `$flink_dist` is url ?
Should run `tar xzf` I think
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3205: [FLINK-5635] Improve Docker tooling to make it eas...
Posted by tonycox <gi...@git.apache.org>.
Github user tonycox commented on a diff in the pull request:
https://github.com/apache/flink/pull/3205#discussion_r100148697
--- Diff: flink-contrib/docker-flink/Dockerfile ---
@@ -36,22 +31,25 @@ ENV PATH $PATH:$FLINK_HOME/bin
EXPOSE 8081
EXPOSE 6123
+# flink-dist can point to a directory, a tarball on the local system, or a url to a tarball
+ARG flink_dist=NOT_SET
+
# Install build dependencies and flink
+ADD $flink_dist $FLINK_INSTALL_PATH
RUN set -x && \
mkdir -p $FLINK_INSTALL_PATH && \
--- End diff --
place of error. when you add `$flink_dist` path of `$FLINK_INSTALL_PATH` will create. I propose to delete making dir here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3205: [FLINK-5635] Improve Docker tooling to make it easier to ...
Posted by jgrier <gi...@git.apache.org>.
Github user jgrier commented on the issue:
https://github.com/apache/flink/pull/3205
Thanks @ex00. I missed that. Actually there is a new Flink on Docker stub in the top-level docs and I was planning to document these scripts there, however we may want to keep this here and just link to it. Anyway, I will update the docs. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3205: [FLINK-5635] Improve Docker tooling to make it easier to ...
Posted by ex00 <gi...@git.apache.org>.
Github user ex00 commented on the issue:
https://github.com/apache/flink/pull/3205
@jgrier thanks for your reply!
I tried to build flink image from your PR and i got next error:
>tar: invalid magic
tar: short read
The command '/bin/sh -c set -x && mkdir -p $FLINK_INSTALL_PATH && apk --update add --virtual build-dependencies curl &&
curl -s $(curl -s https://www.apache.org/dyn/closer.cgi\?preferred\=true)flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSIO
N}-bin-hadoop${HADOOP_VERSION}-scala_${SCALA_VERSION}.tgz | tar xvz -C $FLINK_INSTALL_PATH && ln -s $FLINK_INSTALL_PATH/
flink-$FLINK_VERSION $FLINK_HOME && addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && chown -R flin
k:flink $FLINK_INSTALL_PATH/flink-$FLINK_VERSION && chown -h flink:flink $FLINK_HOME && sed -i -e "s/echo \$mypid >> \$p
id/echo \$mypid >> \$pid \&\& wait/g" $FLINK_HOME/bin/flink-daemon.sh && apk del build-dependencies && rm -rf /var/cache
/apk/*' returned a non-zero code: 1
full error [here](https://gist.github.com/ex00/5246c1df60e51c3a3581eaa6bc683176)
My env:
```
$ docker version
Client:
Version: 1.11.1
API version: 1.23
Go version: go1.5.4
Git commit: 5604cbe
Built: Tue Apr 26 23:44:17 2016
OS/Arch: windows/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:20:08 2016
OS/Arch: linux/amd64
```
Did you have look like error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3205: [FLINK-5635] Improve Docker tooling to make it easier to ...
Posted by ex00 <gi...@git.apache.org>.
Github user ex00 commented on the issue:
https://github.com/apache/flink/pull/3205
Hi @jgrier. Thaks for your PR!
Could you update [README.md](https://github.com/apache/flink/blob/master/flink-contrib/docker-flink/README.md) in accordance with your changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3205: [FLINK-5635] Improve Docker tooling to make it easier to ...
Posted by ex00 <gi...@git.apache.org>.
Github user ex00 commented on the issue:
https://github.com/apache/flink/pull/3205
Really I didn't build Flink. Thanks @jgrier for your comment. My case has been showing that documentation is needed :)
I think next information will be useful for users who want run flink on docker machine of windows.
Then I built image, i tried run docker-compose and I got next message
```
$ docker-compose up -d
Creating dockerflink_jobmanager_1
ERROR: for jobmanager Container command 'docker-entrypoint.sh' not found or does not exist.
```
I tried execute docker-entrypoint.sh manually in container
```
bash-4.3$ /opt/flink/bin/docker-entrypoint.sh
bash: /opt/flink/bin/docker-entrypoint.sh: /bin/sh^M: bad interpreter: No such file or directory
```
I changed line end format for docker-entrypoint.sh on Unix and after that docker-compose has started successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---