You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by merrimanr <gi...@git.apache.org> on 2018/03/26 13:48:02 UTC
[GitHub] metron pull request #973: METRON-1356: Add a mechanism in Java for discoveri...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/973
METRON-1356: Add a mechanism in Java for discovering service host/ports
## Contributor Comments
This PR adds a way for integration tests to lookup the docker machine ip address if there is one. Due to networking limitations in Mac OSX, Docker requires containers be run in a virtual machine (or docker machine). This means that services must be addressed through the virtual machine ip address rather than just localhost. This utility allows integration tests to run in either a typical Docker deployment or when debugging locally in Mac OSX. An example of how it's used is included in the metron-elasticsearch integration tests.
A docker-machine ip address is returned if the following conditions are met:
- docker-machine is installed locally
- a machine called metron-machine exists
- a proper ip address is found for that machine
Otherwise (as in the case of travis) localhost is returned. A method for looking up the correct port for a service is also included in case any ports need to be mapped to different ports within a docker machine.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron docker-ip-lookup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/973.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #973
----
commit a5fc2a1aa11616591227e8ce950e672df2694698
Author: merrimanr <me...@...>
Date: 2018-02-20T22:13:25Z
Merge remote-tracking branch 'mirror/master' into feature/METRON-1344-test-infrastructure
# Conflicts:
# .travis.yml
# metron-interface/metron-alerts/e2e/utils/e2e_util.ts
# metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
# metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
commit 14a573f7743c9be92f852284a1a859c24cb117ae
Author: merrimanr <me...@...>
Date: 2018-02-22T17:31:59Z
initial commit
commit 09c8193b2dc71d37f3e1730f83053e3f196b2d2b
Author: merrimanr <me...@...>
Date: 2018-02-22T18:06:54Z
wait for ES container to come up
commit 1b40e51ca89728647b668e051c67ac1e21abb367
Author: merrimanr <me...@...>
Date: 2018-02-22T20:20:00Z
Merge remote-tracking branch 'mirror/feature/METRON-1344-test-infrastructure' into METRON-1355
# Conflicts:
# .travis.yml
# metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
# metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
commit 2c259543ef59492b1ed393d5fbb8fbf49b36d8c5
Author: merrimanr <me...@...>
Date: 2018-03-15T21:16:10Z
Added documentation for running integration tests locally
commit 8c16377239c9f529e0d3eaa9251de1a7741ed20a
Author: merrimanr <me...@...>
Date: 2018-03-15T21:41:19Z
testing travis
commit 88011461d501bfd0532886f89b44935a11ff3088
Author: merrimanr <me...@...>
Date: 2018-03-15T22:14:02Z
add docker util call and undo travis test
commit 77c0ec5aafe57ae51a0b9677c6b041962bca6607
Author: merrimanr <me...@...>
Date: 2018-03-15T22:19:35Z
add docker util call and undo travis test
commit b66bfe0e7b9c42a8c0da4e43152f5669a6440ed4
Author: merrimanr <me...@...>
Date: 2018-03-16T17:20:12Z
added method to lookup Docker container ports
commit 412c36f167b9ab442e423284981eca1d4f249668
Author: merrimanr <me...@...>
Date: 2018-03-26T13:29:32Z
Merge remote-tracking branch 'mirror/feature/METRON-1344-test-infrastructure' into docker-ip-lookup
# Conflicts:
# metron-contrib/metron-docker-e2e/compose/docker-compose.yml
# metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/utils/ElasticsearchTestUtils.java
----
---
[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/973
@ottobackwards just updated the README. You previously had to lookup the docker machine ip address and substitute it in source code to get the tests to pass if running locally. The text I removed in the latest commit described how to do that. So for testing this, you should no longer have to do that and the tests should just work.
I haven't heard of that before but I will check it out.
---
[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/973
- Followed the steps ( we need a docker cheat sheet for deleting existing old machine/containers btw )
- ran the install/integration-test
not sure it is related:
```bash
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:3.0.2:jar (default-jar) on project metron-hbase-client: You have to use a classifier to attach supplemental artifacts to the project instead of replacing them. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :metron-hbase-client
```
---
[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/973
@merrimanr this looks great. A couple of things:
1. Can you give some steps on how to test this? I would like to keep the FB moving
2. have you ever looked at https://www.testcontainers.org/?
---
[GitHub] metron pull request #973: METRON-1356: Add a mechanism in Java for discoveri...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/metron/pull/973
---
[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/973
Going to close this for now until we figure out what our strategy is going to be with UI e2e tests. We may decide not to include those as part of the travis build and instead rely on unit tests. The e2e tests would then be run manually against full dev.
---