You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gearpump.apache.org by karol-brejna-i <gi...@git.apache.org> on 2017/02/27 13:58:26 UTC
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
GitHub user karol-brejna-i opened a pull request:
https://github.com/apache/incubator-gearpump/pull/162
[GEARPUMP-281] Using new version of docker images for integration tests
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the commit message is formatted like:
`[GEARPUMP-<Jira issue #>] Meaningful description of pull request`
- [ ] Make sure tests pass via `sbt clean test`.
- [ ] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality.
Changes include:
* adding docker project to integrationtest/docker (java, hadoop, kafka, gearpump-launcher)
* updating gearpump-launcher to include new version of java
* update the code to use new images
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/karol-brejna-i/incubator-gearpump GEARPUMP-281
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/162.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #162
----
commit 3cfdd6b3448be77ffd94c0aec27941c5d53d6bf4
Author: karol brejna <ka...@intel.com>
Date: 2017-02-23T18:51:57Z
[GEARPUMP-281] Using new version of docker images for integration tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by karol-brejna-i <gi...@git.apache.org>.
Github user karol-brejna-i commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103432650
--- Diff: integrationtest/docker/kafka/LICENSE ---
@@ -0,0 +1,201 @@
+Apache License
--- End diff --
added licence header to dockerfiles
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by karol-brejna-i <gi...@git.apache.org>.
Github user karol-brejna-i commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103432537
--- Diff: integrationtest/docker/kafka/README.md ---
@@ -0,0 +1,89 @@
+Kafka in Docker
+===
+
+**This is a fork spotify/kafka image. We decided to make it, because the project does not maintain proper tagging.
+For now our tests focus on Kafka 0.8.x, so here is a version that supports Kafka 0.8.x.
+For latest version of kafka image go to the original project.**
+
+This repository provides everything you need to run Kafka in Docker.
+
+For convenience also contains a packaged proxy that can be used to get data from
+a legacy Kafka 7 cluster into a dockerized Kafka 8.
+
+Why?
+---
+The main hurdle of running Kafka in Docker is that it depends on Zookeeper.
+Compared to other Kafka docker images, this one runs both Zookeeper and Kafka
+in the same container. This means:
+
+* No dependency on an external Zookeeper host, or linking to another container
+* Zookeeper and Kafka are configured to work together out of the box
+
+Run
+---
+
+```bash
+docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 grubykarol/kafka
--- End diff --
No, we use it without it. It's just example from original README.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #162: [GEARPUMP-281] Using new version of docker im...
Posted by codecov-io <gi...@git.apache.org>.
Github user codecov-io commented on the issue:
https://github.com/apache/incubator-gearpump/pull/162
# [Codecov](https://codecov.io/gh/apache/incubator-gearpump/pull/162?src=pr&el=h1) Report
> Merging [#162](https://codecov.io/gh/apache/incubator-gearpump/pull/162?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gearpump/commit/8cd0f0c9b31944d892972178365871b8134b1619?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #162 +/- ##
======================================
Coverage 71.5% 71.5%
======================================
Files 190 190
Lines 6054 6054
Branches 523 523
======================================
Hits 4329 4329
Misses 1725 1725
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gearpump/pull/162?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `\u0394 = absolute <relative> (impact)`, `� = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gearpump/pull/162?src=pr&el=footer). Last update [8cd0f0c...3cfdd6b](https://codecov.io/gh/apache/incubator-gearpump/compare/8cd0f0c9b31944d892972178365871b8134b1619...3cfdd6b3448be77ffd94c0aec27941c5d53d6bf4?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by karol-brejna-i <gi...@git.apache.org>.
Github user karol-brejna-i commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103432669
--- Diff: integrationtest/docker/README.md ---
@@ -1,28 +1,23 @@
-# Gearpump Launcher Docker Image
-
-The image helps developer to setup/test a local [Gearpump](https://github.com/apache/incubator-gearpump.git) cluster quickly. The image is based on a minimal JRE8 environment with Python support.
-
-## Usage
-
-Here are the commands to launch a cluster. You can launch as many worker containers as you wish but only one master for the time being.
-```
-export GEARPUMP_HOME=/path/to/gearpump
-
-docker run -d \
- -h master0 --name master0 \
- -v $GEARPUMP_HOME:/opt/gearpump \
- -e JAVA_OPTS=-Dgearpump.cluster.masters.0=master0:3000 \
- -p 8090:8090 \
- stanleyxu2005/gearpump-launcher \
- master -ip master0 -port 3000
-
-docker run -d \
- --link master0 \
- -v $GEARPUMP_HOME:/opt/gearpump \
- -e JAVA_OPTS=-Dgearpump.cluster.masters.0=master0:3000 \
- stanleyxu2005/gearpump-launcher \
- worker
-
-docker exec master0 gear info
-docker exec master0 gear app -jar /path/to/userapp.jar [mainclass] [args]
-```
+This folder contains docker images definitions used in integration tests of Gearpump.
+
+
+These include:
+
+ * [The standalone single node Kafka cluster with Zookeeper](/kafka)
+ * [The Hadoop image](/hadoop)
+ * [The Gearpump Cluster Launcher and Storm Client](/gearpump)
+ * [Java 8 JRE image](/java)
+
+
+We decided to fork spotify/kafka image, because the project does not maintain proper tagging.
+For now our tests focus on Kafka 0.8.x, and the project does not support this version.
+
+Hadoop docker image (https://hub.docker.com/r/sequenceiq/hadoop-docker/) is well maintained.
+We rely on version 2.6.0 and we feel there is no need to duplicate it.
+
+Gearpump Cluster Launcher helps developer to setup/test a local Gearpump cluster quickly.
+The image is based on a minimal JRE8 environment with Python support.
+
+We used to base Gearpump Cluster Launcher on errordeveloper/oracle-jre image but it stuck on version 8u66-b17 and doesn't support tagging.
+We also probably hit a Java bug (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133205) that requires never version of JRE.
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103382701
--- Diff: integrationtest/docker/README.md ---
@@ -1,28 +1,23 @@
-# Gearpump Launcher Docker Image
-
-The image helps developer to setup/test a local [Gearpump](https://github.com/apache/incubator-gearpump.git) cluster quickly. The image is based on a minimal JRE8 environment with Python support.
-
-## Usage
-
-Here are the commands to launch a cluster. You can launch as many worker containers as you wish but only one master for the time being.
-```
-export GEARPUMP_HOME=/path/to/gearpump
-
-docker run -d \
- -h master0 --name master0 \
- -v $GEARPUMP_HOME:/opt/gearpump \
- -e JAVA_OPTS=-Dgearpump.cluster.masters.0=master0:3000 \
- -p 8090:8090 \
- stanleyxu2005/gearpump-launcher \
- master -ip master0 -port 3000
-
-docker run -d \
- --link master0 \
- -v $GEARPUMP_HOME:/opt/gearpump \
- -e JAVA_OPTS=-Dgearpump.cluster.masters.0=master0:3000 \
- stanleyxu2005/gearpump-launcher \
- worker
-
-docker exec master0 gear info
-docker exec master0 gear app -jar /path/to/userapp.jar [mainclass] [args]
-```
+This folder contains docker images definitions used in integration tests of Gearpump.
+
+
+These include:
+
+ * [The standalone single node Kafka cluster with Zookeeper](/kafka)
+ * [The Hadoop image](/hadoop)
+ * [The Gearpump Cluster Launcher and Storm Client](/gearpump)
+ * [Java 8 JRE image](/java)
+
+
+We decided to fork spotify/kafka image, because the project does not maintain proper tagging.
+For now our tests focus on Kafka 0.8.x, and the project does not support this version.
+
+Hadoop docker image (https://hub.docker.com/r/sequenceiq/hadoop-docker/) is well maintained.
+We rely on version 2.6.0 and we feel there is no need to duplicate it.
+
+Gearpump Cluster Launcher helps developer to setup/test a local Gearpump cluster quickly.
+The image is based on a minimal JRE8 environment with Python support.
+
+We used to base Gearpump Cluster Launcher on errordeveloper/oracle-jre image but it stuck on version 8u66-b17 and doesn't support tagging.
+We also probably hit a Java bug (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133205) that requires never version of JRE.
--- End diff --
typo 'nerver' => 'newer'
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/162
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #162: [GEARPUMP-281] Using new version of docker im...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/162
+1 Merging
Thanks Karol!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #162: [GEARPUMP-281] Using new version of docker im...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/162
Hi Karol, would you mind do some code modification in org.apache.gearpump.integrationtest.Docker.scala, change the implementation of `getNetworkGateway` to
```
final def getNetworkGateway(container: String): String = {
trace(container, s"Get gateway of container...") {
Docker.inspect(container, "--format={{.NetworkSettings.Gateway}}")
}
}
```
Then rerun the integration test on your Ubuntu environment to see whether the modification is OK?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103349793
--- Diff: integrationtest/docker/kafka/LICENSE ---
@@ -0,0 +1,201 @@
+Apache License
--- End diff --
We don't keep a separate license file for sub-folders. Instead, we add license headers to source files (except README).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103385452
--- Diff: integrationtest/docker/kafka/README.md ---
@@ -0,0 +1,89 @@
+Kafka in Docker
+===
+
+**This is a fork spotify/kafka image. We decided to make it, because the project does not maintain proper tagging.
+For now our tests focus on Kafka 0.8.x, so here is a version that supports Kafka 0.8.x.
+For latest version of kafka image go to the original project.**
+
+This repository provides everything you need to run Kafka in Docker.
+
+For convenience also contains a packaged proxy that can be used to get data from
+a legacy Kafka 7 cluster into a dockerized Kafka 8.
+
+Why?
+---
+The main hurdle of running Kafka in Docker is that it depends on Zookeeper.
+Compared to other Kafka docker images, this one runs both Zookeeper and Kafka
+in the same container. This means:
+
+* No dependency on an external Zookeeper host, or linking to another container
+* Zookeeper and Kafka are configured to work together out of the box
+
+Run
+---
+
+```bash
+docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 grubykarol/kafka
--- End diff --
it needs docker-machine?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by karol-brejna-i <gi...@git.apache.org>.
Github user karol-brejna-i commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103432682
--- Diff: integrationtest/docker/java/README.md ---
@@ -0,0 +1,18 @@
+A minimalistic Oracle JDK 8 container on top of busybox.
+
+We used to base Gearpump Cluster Launcher on errordeveloper/oracle-jre image but it stuck on version 8u66-b17 and doesn't support tagging.
+We also probably hit a Java bug (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133205) that requires never version of JRE.
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103382851
--- Diff: integrationtest/docker/java/README.md ---
@@ -0,0 +1,18 @@
+A minimalistic Oracle JDK 8 container on top of busybox.
+
+We used to base Gearpump Cluster Launcher on errordeveloper/oracle-jre image but it stuck on version 8u66-b17 and doesn't support tagging.
+We also probably hit a Java bug (http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133205) that requires never version of JRE.
--- End diff --
also has typo here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #162: [GEARPUMP-281] Using new version of docker im...
Posted by karol-brejna-i <gi...@git.apache.org>.
Github user karol-brejna-i commented on the issue:
https://github.com/apache/incubator-gearpump/pull/162
@huafengw no problem. Let me get back to you soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #162: [GEARPUMP-281] Using new version of do...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/162#discussion_r103349524
--- Diff: integrationtest/docker/java/README.md ---
@@ -0,0 +1,18 @@
+A minimalistic Oracle JDK 8 container on top of busybox.
--- End diff --
Great investigation and README
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #162: [GEARPUMP-281] Using new version of docker im...
Posted by karol-brejna-i <gi...@git.apache.org>.
Github user karol-brejna-i commented on the issue:
https://github.com/apache/incubator-gearpump/pull/162
The change you suggested works fine on my ubuntu 14.04:
```
[info] Run completed in 25 minutes, 45 seconds.
[info] Total number of tests run: 66
[info] Suites: completed 9, aborted 0
[info] Tests: succeeded 66, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---