You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by merrimanr <gi...@git.apache.org> on 2017/12/06 14:47:07 UTC
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/858
METRON-1344: Externalize the infrastructural components using integration tests
## Contributor Comments
This PR will add infrastructure to our Travis build that will allow the Alerts UI e2e test to be run. There are several outstanding issues that still need to be worked out so DO NOT MERGE this yet.
This is a first pass at a potential Docker-based solution and is meant to be a POC. The intention is to facilitate further discussion around the general approach and provide a working example that we can build off of.
A good place to start is the .travis file. This provides a good guide on how this infrastructure is spun up. It is assumed Docker and Docker Compose are installed. To use outside of travis in a local dev environment (assuming Mac OSX):
1. Build Metron with `mvn clean install -DskipTests`
1. Create a Docker machine with `metron-contrib/metron-docker/scripts/create-docker-machine.sh`
1. Set the Docker env variables with `eval $(docker-machine env metron-machine)`
1. Build the base Metron image that installs Java: `docker build ./metron-centos/ -t "metron-centos"`
1. Spin up the environment: `cd metron-contrib/metron-docker-e2e/compose && docker-compose up -d`
A working environment should now be available. To verify get the Docker machine address with `echo $DOCKER_HOST`. Services should be available on the ports specified in `metron-contrib/metron-docker-e2e/compose/docker-compose.yml`. For example, assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should be available at http://192.168.99.100:9210.
At this point only a single e2e test is run due to the significant refactoring being done in https://github.com/apache/metron/pull/857.
Here are my thoughts so far based on work in this PR:
- The Docker environment creation and startup adds about 3 minutes to the build. This is about what I expected and hopefully we can get this time back as we move other tests to a reusable environment.
- I'm not convinced spinning up containers for the Alerts UI and REST are necessary or desired. We may be able to cut some time off the build by running them directly on the Travis host instead of in Docker
- I experimented with caching the Docker images but pulling them every time was actually faster
There is still significant work to be done including:
- I was able to get the full suite of tests to run successfully in previous commits but have all but e2e tests commented out for now to make it easier to see what's going on in Travis. These will need to be added back once we get closer to a final solution.
- Once the e2e test refactoring is done those changes will need merged in and tested
- The license check is commented out right now because I inadvertently added some new dependency versions (have no idea why). Still need to track that down.
- There is an intermittent error that happens when starting up REST. Still working on tracking this down.
Looking forward to some feedback.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron e2e-in-travis
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/858.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #858
----
commit b8fb307748c4e83e88cd9c156464eb99b5b13025
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:05:14Z
initial commit
commit faabe6789cb83abe6d5ba6005fa10f73225e253a
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:08:32Z
travis test
commit c61ca37491225df77c14363c499c7465bca15651
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:09:55Z
travis test
commit dda0d7a6d7264dbfcbfb8bdc6e6bf2fc3325ecdb
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:15:31Z
travis test
commit 083f4290b8a3129e27bcca15879bf3889361357a
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:19:20Z
travis test
commit 2b4a90e777756d24d1e74ec80cfbf74832f8d648
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:56:57Z
travis test
commit 7d9b47f5de63ceec66891eb4ee1e5122f908a2a9
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T20:18:32Z
travis test
commit 594d94c754ca6fc0e62a3fa82505598b0e71d7ae
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T20:38:49Z
travis test
commit cd06db23494f084abb05a3adbbb2b3170927aaf8
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:11:01Z
travis test
commit 75a8d13c4ecc07a69443aaa61d4ba4859f860946
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:17:22Z
travis test
commit cbf2da7597514a3c8e02309f463d8726d84c13be
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:49:07Z
travis test
commit 280fdf5bca481049ae0f83c5deae1f93611959fa
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:25:20Z
travis test
commit 0ef3f2a79824942edace28b57bf93975760177f6
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:31:03Z
travis test
commit b21ce8288bbbabd75e4eb2e877de20db81fe3b8d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:36:16Z
travis test
commit 4e741903281875f31ccee0c5d90868a329ff2a67
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:40:23Z
travis test
commit 2e6d74ac80ccc40f3f96e29640d5e3b07eba870d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:47:12Z
travis test
commit f3cb688563fc201ea4178a45369864184a176dec
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:51:06Z
travis test
commit 55ca067f3768f88f7dade0602a00af92f48e7616
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:56:04Z
travis test
commit a3c65db085744fc3b65491211a769cf19a0695e9
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:15:29Z
travis test
commit 4751848e7cf5221976e7bae23cea8fc9ee75a72f
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:18:16Z
travis test
commit 86a0a5db301b0246ae650d67e1e45c4a442ec074
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:27:23Z
travis test
commit ce2c75acbff2486f028543ef2a56ee3f21a58fc0
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:36:12Z
travis test
commit b5c5a94ef53bc154019c623d7a8368ff99950f9d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:48:56Z
travis test
commit ce2bf87199bfc5fb2db1c7d1df7c07ff992b8905
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T00:04:45Z
travis test
commit b426a28cc49e21db4907db5c35268c193e69a8da
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T00:49:24Z
travis test
commit 09aa8f080c6ccf3c4e9d5fb267fa86f113ee286c
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T01:22:07Z
travis test
commit 1e89a1dceea81c25411f596e29cb73964a408b17
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T01:23:36Z
travis test
commit 02c147bcc9543f3bc07c91e06dc11c9b35b3e422
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:15:21Z
travis test
commit a693cf95398e8300900530b085c4724b8a822026
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:21:58Z
travis test
commit 39d1d96b497b67b40e1aabed23c84a81ec3f009d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:54:40Z
travis test
----
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
I created a feature branch called "feature/METRON-1344-test-infrastructure" and switched the base of this PR to that instead of master. What is the next step?
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r156183357
--- Diff: metron-interface/metron-alerts/protractor.conf.js ---
@@ -25,27 +25,28 @@ var SpecReporter = require('jasmine-spec-reporter').SpecReporter;
exports.config = {
allScriptsTimeout: 25000,
specs: [
- './e2e/login/login.e2e-spec.ts',
- './e2e/alerts-list/alerts-list.e2e-spec.ts',
+ // './e2e/login/login.e2e-spec.ts',
--- End diff --
Similar comment to the above about removing vs commenting out code.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/metron/pull/858
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
@ottobackwards I also added a way to run the e2e tests locally. Once your Docker app is up and running locally, get the $DOCKER_HOST ip address and replace "localhost" with that ip address in `metron-interface/metron-alerts/protractor.conf.js` (should be 3 places you have to update). Mine looks something like this:
```
exports.config = {
...
baseUrl: 'http://192.168.99.100:4201/',
...
params: {
rest: {
url: '192.168.99.100:8082'
},
elasticsearch: {
url: '192.168.99.100:9210'
}
}
}
```
Then you can run the e2e tests with `cd metron-interface/metron-alerts && npm run e2e`
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
As long as breakpoints are limited to code we maintain in Metron, I think it's just a matter of documenting how to set it up. For example, I run REST locally in Intellij and debugging works fine against the in memory components or full dev. I think the Storm topologies should continue to be run locally with LocalCluster so we're covered there too as along as the topologies can access the services running in Docker. The only difference I can think of would be that now you can't put a breakpoint in Kafka or Elasticsearch server side code without enabling remote debugging.
I have no problem with this being a feature branch. There is a lot of work to be done and design issues to work through. The tradeoff of having this in a feature branch is that it will delay the ability to run the UI e2e tests in our build. The tradeoff of doing it incrementally is that our build times will increase until we switch the tests over because we will have to use both in memory components and Docker. I'm happy to do it either way.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
Sure a discussion on that topic is fine with me. I'm referring to the existing integration tests.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r156182999
--- Diff: metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts ---
@@ -47,7 +47,7 @@ describe('metron-alerts configure table', function() {
let newColNamesColumnConfig = [ 'score', 'timestamp', 'source:type', 'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
'ip_dst_addr', 'host', 'alert_status', 'guid' ];
- page.clearLocalStorage();
+ //page.clearLocalStorage();
--- End diff --
Can we remove this line instead of commenting it out?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/858
We would want to remote debug any code that we maintain in Metron, so any services which run metron code would need to have remote debugging (e.g. storm, the REST app). Ironically, this is one of those situations where just running the in-memory components in a separate process would be easier, since it's just a JVM param to pass in to enable remote debugging.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/858
This smells like it might be a bigger effort than one PR, I think I'd rather have smaller PRs on a feature branch so each is easier to review. I would expec that we will want to try it out and live with it a bit as it will impact day-to-day productivity fairly fundamentally. I'd vote for a feature branch, but I'd like to hear other people's impressions.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r155339646
--- Diff: metron-interface/metron-alerts/pom.xml ---
@@ -130,4 +146,44 @@
</plugin>
</plugins>
</build>
+ </profile>
+
+ <profile>
+ <id>e2e</id>
+ <build>
+ <plugins>
+ <plugin>
--- End diff --
Once we get closer to consensus on what this is going to look like, I promise I will add extensive documentation.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
Doesn't necessarily have to be Docker but yes you would have to start something up. That step is now outside of the integration tests.
We could keep the current in memory code and offer a debug mode. Is there a benefit to using an in-memory component when debugging?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr, I have created labels in jira that are applicable to feature branch Jiras,
This will allow for tracking these jiras in confluence as I have done here : https://cwiki.apache.org/confluence/display/METRON/Metron+Extension+System+and+Parser+Extensions.
I would suggest creating a page for this feature branch were we can document things in this area, and using the labels on the liras
metron-feature-canidate, {create a metron-feature-externalize-? label}
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/858
If we think that remote debugging would cause this to balloon, should we consider making this a feature branch so multiple PRs with iterative solutions can exist and be reviewed separately? Just a thought.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
Any chance we can have a discuss on moving other integration tests over? You mean the regular tests right, like profiler etc?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/858
Regarding remote debugging, it seems that having both in memory components AND docker is the worst of all worlds here. I would prefer to not live in a world where we can't simplify this to one solution rather than having to continue to maintain both. I would vote for either enabling remote debugging in docker *xor* just migrating the in memory components out and documenting remote debugging for those processes. Either way, documentation should be part of this PR.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
We will, if this pr goes through need to have a 'writing new integration tests' readme about namespacing etc..
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
@cestella are you good with merging this in to the feature branch? I believe I addressed your comments.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
OK,
After getting the new changes, everything works as advertised. I was able to run on mac os , and the end to end passed. I ran a few times in a row and there where no issues, so it looks like setup/teardown works fine.
I think we you have done to this point is great @merrimanr, really great work. I like the speed of getting compose up and everything.
The idea of doing the namespaces for ES is great too.
I did not run the integration test
To summarize my feedback:
- I would like some scripts that wrap this up into high level usage
- I think we need doc around writing tests that use this framework ( will we be adding tests one by one + any new tests )?
- I think we should enable debugging on the services in docker on known ports. Knowing that to @justinleet 's point, any java dev. can get their tool of choice to work. We can do intellij specific examples on top of that as well
- I think we should have a print out statement, similar to what we have at the end of vagrant up/ansible that says the ip addresses and ports exposed + service name. this would include the debugging. I know this helps with vagrant, it would help here.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
I agree with the discuss thread, I'll leave comment on the list for there
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr Did you look into what migrating our non-e2e tests integration tests would involve? I think for a POC, it's important to get a sense of how we'd be able to unify the infrastructure of these, even if they aren't necessarily all unified at POC time. If not, are you planning on investigating that once you resolve the current list of e2e issues?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr can you add what directory you should be in when you run the commands?
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r155288298
--- Diff: metron-interface/metron-alerts/e2e/utils/e2e_util.ts ---
@@ -46,25 +48,53 @@ export function waitForStalenessOf (_element ) {
}
export function loadTestData() {
--- End diff --
I think it is important that these test functions are commented and documented. Not only for maintaining them, but they will possibly be examples for other efforts.
Some of these functions around setting up tests and tearing them down would themselves be useful as utility functions more general than the end to end possibly
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
I agree with everything that has been said. I will address the minor changes @cestella suggested and start the discussion thread. Will METRON-1344 work as the base Jira for the feature branch?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr would you imagine someone just leaving this running all the time? or starting on demand per use?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr I am +1 on getting this down to the feature branch and moving on.
Due to the way the feature branch works, I think it is OK to do so if @cestella doesn't get back in time. It's been two weeks.
Let's get this moving
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
No matter what approach we take, my point is that we need to have some thing that says you need to run docker to debug the tests. Beyond that, we don't have to over back this. I think the approach in this pr is preferable to the in-memory components in separate processes.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
Sorry @ottobackwards, you're right the instructions are a little vague. First, `metron-contrib/metron-docker-e2e/compose` is the directory you should be in. I should have added that at the beginning. Not all steps require you be in that directory but steps 4 and 5 do.
So you can't run the tests in the same exact way now because the Alerts UI host is hardcoded in the e2e tests (add that to our list of things to address). It is hardcoded to "localhost" which works fine in travis but won't work in Mac OSX because of networking limitations (Docker has to run in a vagrant machine). Let me come up with a plan that will allow you to easily run the e2e tests locally.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
Wow, this is awesome. I'm looking forward to getting into it. First thing that pops to mind is how this relates to or is effected by @nickwallen's https://github.com/apache/metron/pull/854
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r156215355
--- Diff: metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts ---
@@ -47,7 +47,7 @@ describe('metron-alerts configure table', function() {
let newColNamesColumnConfig = [ 'score', 'timestamp', 'source:type', 'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
'ip_dst_addr', 'host', 'alert_status', 'guid' ];
- page.clearLocalStorage();
+ //page.clearLocalStorage();
--- End diff --
Done
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/858
I'd actually recommend not using METRON-1344 as the base JIRA. I'd recommend creating a new JIRA where the requirements for the end-state are laid out in the description. I'd call this one a PoC JIRA that the base JIRA depends on.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/metron/pull/858
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r156215392
--- Diff: metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts ---
@@ -58,26 +58,26 @@ describe('metron-alerts configure table', function() {
page.saveConfigureColumns();
});
- it('should rename columns from table configuration', () => {
- page.clearLocalStorage();
- page.navigateTo();
-
- page.clickConfigureTable();
- page.renameColumn('enrichments:geo:ip_dst_addr:country', 'Country');
- page.saveConfigureColumns();
-
- page.clickTableText('FR');
- expect(page.getSearchText()).toEqual('Country:FR');
- expect(page.getChangesAlertTableTitle('Alerts (169)')).toEqual('Alerts (25)');
- page.clickClearSearch();
-
- expect(page.getChangesAlertTableTitle('Alerts (25)')).toEqual('Alerts (169)');
- page.setSearchText('Country:FR');
- expect(page.getChangesAlertTableTitle('Alerts (169)')).toEqual('Alerts (25)');
- page.clickClearSearch();
-
- expect(page.getTableColumnNames()).toContain('Country', 'for renamed column names for alert list table');
-
- });
+ // it('should rename columns from table configuration', () => {
--- End diff --
Done
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
So, debugging the integration tests will require having docker running in the background now?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
@ottobackwards I don't see anything obviously wrong. Maybe you could try pulling the latest commits or copying the one here: https://github.com/merrimanr/incubator-metron/blob/e2e-in-travis/metron-interface/metron-alerts/protractor.conf.js. Looks like your docker machine host is the same.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/858
Yep, I think a JIRA for the feature branch and each PR as a subtask is a good idea.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r155288566
--- Diff: metron-interface/metron-alerts/pom.xml ---
@@ -130,4 +146,44 @@
</plugin>
</plugins>
</build>
+ </profile>
+
+ <profile>
+ <id>e2e</id>
+ <build>
+ <plugins>
+ <plugin>
--- End diff --
should we document with a comment what these plugins are for?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr my protractor config does not have a params entry, so I have added it as you have it there. Is that usual?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr I'm not sure I understand your steps above, do they run the tests or is that separate?
What is step 6?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
Ok, I'm going to try running through some things.
One point, and I will admit this just may be me:
I see the same problem with this as I had with the original docker stuff.
The list of instructions for initial use is there, but I don't quite understand how after this initial use I would go back to this setup and use it day to day. Do these steps need to happen every time?
Is there a way to script this such that there is an init_metron_testing_docker, start_metron_testing_docker shutdown_metron_testing_docker etc?
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
I would love to have a Debugging readme that is even more comprehensive than just integration tests. Being able to easily debug is important to me and I do it all the time.
I had to do some debugging to convert this test and it was the same process I have used before. Just put a break point in your test and run the test in Debug in Intellij. Debugging Storm is a different animal and I suspect we would continue to use LocalCluster. We would need to ensure the topology running locally in an IDE would still be able to interact with services in Docker. I have successfully done this in the past for each of the topologies.
The only other case I can think of would be putting breakpoints in code of services we use (Kafka code for example). Is that something we want to be able to do? I can see some value although I wouldn't think that's a normal thing.
We could always keep the in memory module and offer a way to start them up in a separate process. That way you could still debug everything in Intellij. I think I would prefer that option over enabling remote debugging in Docker containers.
And again, we don't have to use Docker. We could just move the in memory stuff outside of the integration tests and add a layer to manage them in separate processes. But that means we now have to implement some features Docker provides us with custom code.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
@justinleet latest commit includes the changes I had to make to switch a preexisting integration test to the new infrastructure. I verified it works locally, we'll see how it goes with Travis.
It was pretty straightforward and consisted of mainly 2 steps:
1. Remove the in memory components and replace any helper methods for setup/teardown
2. Add a namespace to all the test assets (indices and doc types in this case)
I believe converting other tests will follow this pattern as well. To answer @ottobackwards's original question about [#854](https://github.com/apache/metron/pull/854), adding namespaces will be required when running the tests in parallel against shared infrastructure. Cleaning up at the beginning/end won't be enough.
This is meant only as an example, happy to revert it and add it in a future PR.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
GitHub user merrimanr reopened a pull request:
https://github.com/apache/metron/pull/858
METRON-1344: Externalize the infrastructural components using integration tests
## Contributor Comments
This PR will add infrastructure to our Travis build that will allow the Alerts UI e2e test to be run. There are several outstanding issues that still need to be worked out so DO NOT MERGE this yet.
This is a first pass at a potential Docker-based solution and is meant to be a POC. The intention is to facilitate further discussion around the general approach and provide a working example that we can build off of.
A good place to start is the .travis file. This provides a good guide on how this infrastructure is spun up. It is assumed Docker and Docker Compose are installed. To use outside of travis in a local dev environment (assuming Mac OSX):
1. Build Metron with `mvn clean install -DskipTests`
1. Create a Docker machine with `metron-contrib/metron-docker/scripts/create-docker-machine.sh`
1. Set the Docker env variables with `eval $(docker-machine env metron-machine)`
1. Build the base Metron image that installs Java: `docker build ./metron-centos/ -t "metron-centos"`
1. Spin up the environment: `cd metron-contrib/metron-docker-e2e/compose && docker-compose up -d`
A working environment should now be available. To verify get the Docker machine address with `echo $DOCKER_HOST`. Services should be available on the ports specified in `metron-contrib/metron-docker-e2e/compose/docker-compose.yml`. For example, assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should be available at http://192.168.99.100:9210.
At this point only a single e2e test is run due to the significant refactoring being done in https://github.com/apache/metron/pull/857.
Here are my thoughts so far based on work in this PR:
- The Docker environment creation and startup adds about 3 minutes to the build. This is about what I expected and hopefully we can get this time back as we move other tests to a reusable environment.
- I'm not convinced spinning up containers for the Alerts UI and REST are necessary or desired. We may be able to cut some time off the build by running them directly on the Travis host instead of in Docker
- I experimented with caching the Docker images but pulling them every time was actually faster
There is still significant work to be done including:
- I was able to get the full suite of tests to run successfully in previous commits but have all but e2e tests commented out for now to make it easier to see what's going on in Travis. These will need to be added back once we get closer to a final solution.
- Once the e2e test refactoring is done those changes will need merged in and tested
- The license check is commented out right now because I inadvertently added some new dependency versions (have no idea why). Still need to track that down.
- There is an intermittent error that happens when starting up REST. Still working on tracking this down.
Looking forward to some feedback.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron e2e-in-travis
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/858.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #858
----
commit b8fb307748c4e83e88cd9c156464eb99b5b13025
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:05:14Z
initial commit
commit faabe6789cb83abe6d5ba6005fa10f73225e253a
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:08:32Z
travis test
commit c61ca37491225df77c14363c499c7465bca15651
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:09:55Z
travis test
commit dda0d7a6d7264dbfcbfb8bdc6e6bf2fc3325ecdb
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:15:31Z
travis test
commit 083f4290b8a3129e27bcca15879bf3889361357a
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:19:20Z
travis test
commit 2b4a90e777756d24d1e74ec80cfbf74832f8d648
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:56:57Z
travis test
commit 7d9b47f5de63ceec66891eb4ee1e5122f908a2a9
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T20:18:32Z
travis test
commit 594d94c754ca6fc0e62a3fa82505598b0e71d7ae
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T20:38:49Z
travis test
commit cd06db23494f084abb05a3adbbb2b3170927aaf8
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:11:01Z
travis test
commit 75a8d13c4ecc07a69443aaa61d4ba4859f860946
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:17:22Z
travis test
commit cbf2da7597514a3c8e02309f463d8726d84c13be
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:49:07Z
travis test
commit 280fdf5bca481049ae0f83c5deae1f93611959fa
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:25:20Z
travis test
commit 0ef3f2a79824942edace28b57bf93975760177f6
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:31:03Z
travis test
commit b21ce8288bbbabd75e4eb2e877de20db81fe3b8d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:36:16Z
travis test
commit 4e741903281875f31ccee0c5d90868a329ff2a67
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:40:23Z
travis test
commit 2e6d74ac80ccc40f3f96e29640d5e3b07eba870d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:47:12Z
travis test
commit f3cb688563fc201ea4178a45369864184a176dec
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:51:06Z
travis test
commit 55ca067f3768f88f7dade0602a00af92f48e7616
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:56:04Z
travis test
commit a3c65db085744fc3b65491211a769cf19a0695e9
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:15:29Z
travis test
commit 4751848e7cf5221976e7bae23cea8fc9ee75a72f
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:18:16Z
travis test
commit 86a0a5db301b0246ae650d67e1e45c4a442ec074
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:27:23Z
travis test
commit ce2c75acbff2486f028543ef2a56ee3f21a58fc0
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:36:12Z
travis test
commit b5c5a94ef53bc154019c623d7a8368ff99950f9d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:48:56Z
travis test
commit ce2bf87199bfc5fb2db1c7d1df7c07ff992b8905
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T00:04:45Z
travis test
commit b426a28cc49e21db4907db5c35268c193e69a8da
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T00:49:24Z
travis test
commit 09aa8f080c6ccf3c4e9d5fb267fa86f113ee286c
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T01:22:07Z
travis test
commit 1e89a1dceea81c25411f596e29cb73964a408b17
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T01:23:36Z
travis test
commit 02c147bcc9543f3bc07c91e06dc11c9b35b3e422
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:15:21Z
travis test
commit a693cf95398e8300900530b085c4724b8a822026
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:21:58Z
travis test
commit 39d1d96b497b67b40e1aabed23c84a81ec3f009d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:54:40Z
travis test
----
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr Is it worth having a new discuss thread where you lay out what you've done, where you expect this to end, and what (if any) work that would be nice to have but isn't essential for this to be feature complete?
i.e. is the endpoint where we merge to master (and I'm injecting my own bias here, so feel free to adjust as you see fit):
1. All e2e and int tests run on common infrastructure.
1. All e2e and int tests are run automatically in the Travis build.
1. All e2e and int tests run repeatably and reliably in the Travis build.
1. Debugging options are available and documented.
1. The new infra and how to interact with it is documented.
1. Old infrastructure removed (anything unused or commented out is deleted, instead of staying).
@ottobackwards @cestella Is that reasonable? I don't really want to make decisions regarding the specific end criteria of a feature branch in the POC PR.
Sidenote, I like the top level PR and the child tasks for this. It'd be a good way to both organize and flesh out the TODO list you had and give everyone an easy way to see where things are (and where they can hop in and help out)
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
I would be for +1 and landing this pr on the feature branch, then we can discuss next steps, proposals and start new pr's against FB
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
```bash
metron-alerts configure table
✗ should select columns from table configuration
- Error: connect ECONNREFUSED 127.0.0.1:9210
- Error: socket hang up
- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
**************************************************
* Failures *
**************************************************
1) metron-alerts configure table should select columns from table configuration
- Error: connect ECONNREFUSED 127.0.0.1:9210
- Error: socket hang up
- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
Executed 1 of 1 spec (1 FAILED) in 1 min 40 secs.
[09:14:23] I/launcher - 0 instance(s) of WebDriver still running
[09:14:23] I/launcher - chrome #01 failed 1 test(s)
[09:14:23] I/launcher - overall: 1 failed spec(s)
[09:14:23] E/launcher - Process exited with error code 1
```
I don't think that my protractor stuff is correct. Here is what I have:
```json
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// Protractor configuration file, see link for more information
// https://github.com/angular/protractor/blob/master/docs/referenceConf.js
/*global jasmine */
var SpecReporter = require('jasmine-spec-reporter').SpecReporter;
exports.config = {
allScriptsTimeout: 25000,
specs: [
// './e2e/login/login.e2e-spec.ts',
// './e2e/alerts-list/alerts-list.e2e-spec.ts',
'./e2e/alerts-list/configure-table/configure-table.e2e-spec.ts',
//'./e2e/alerts-list/save-search/save-search.e2e-spec.ts',
//'./e2e/alerts-list/tree-view/tree-view.e2e-spec.ts',
// './e2e/alerts-list/alert-filters/alert-filters.e2e-spec.ts',
// './e2e/alerts-list/alert-status/alerts-list-status.e2e-spec.ts',
// './e2e/alert-details/alert-status/alert-details-status.e2e-spec.ts',
//'./e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts'
],
capabilities: {
'browserName': 'chrome',
'chromeOptions': {
args: [ "--headless", "--disable-gpu", "--window-size=1435,850" ],
'prefs': {
'credentials_enable_service': false,
'profile': { 'password_manager_enabled': false}
}
}
},
directConnect: true,
baseUrl: 'http://192.168.99.100:4201/',
params: {
rest: {
url: '192.168.99.100:8082'
},
elasticsearch : {
url: '192.168.99.100:9210'
}
},
framework: 'jasmine',
jasmineNodeOpts: {
showColors: true,
defaultTimeoutInterval: 50000,
print: function() {}
},
useAllAngular2AppRoots: true,
rootElement: 'metron-alerts-root',
beforeLaunch: function() {
require('ts-node').register({
project: 'e2e'
});
},
onPrepare: function() {
var createMetaAlertsIndex = require('./e2e/utils/e2e_util').createMetaAlertsIndex;
createMetaAlertsIndex();
jasmine.getEnv().addReporter(new SpecReporter());
setTimeout(function() {
browser.driver.executeScript(function() {
return {
width: window.screen.availWidth,
height: window.screen.availHeight
};
}).then(function(result) {
browser.driver.manage().window().setSize(result.width, result.height);
});
});
},
onComplete: function() {
var createMetaAlertsIndex = require('./e2e/utils/e2e_util').createMetaAlertsIndex;
createMetaAlertsIndex();
}
};
```
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
Point taken on maintaining 2 different solutions. Which services would people want to enable remote debugging on? I've only ever debugged Metron code so I don't know which services are important to people or why you would need to put breakpoints in core service code. Enabling remote debugging will likely be a little bit different for each service since each project has their own mechanism for adding extra jvm params. I'm happy to do a POC for this once I understand which services are important.
@ottobackwards I usually just leave mine running but spinning it up/down is really fast once all the images are loaded.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r156183048
--- Diff: metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts ---
@@ -58,26 +58,26 @@ describe('metron-alerts configure table', function() {
page.saveConfigureColumns();
});
- it('should rename columns from table configuration', () => {
- page.clearLocalStorage();
- page.navigateTo();
-
- page.clickConfigureTable();
- page.renameColumn('enrichments:geo:ip_dst_addr:country', 'Country');
- page.saveConfigureColumns();
-
- page.clickTableText('FR');
- expect(page.getSearchText()).toEqual('Country:FR');
- expect(page.getChangesAlertTableTitle('Alerts (169)')).toEqual('Alerts (25)');
- page.clickClearSearch();
-
- expect(page.getChangesAlertTableTitle('Alerts (25)')).toEqual('Alerts (169)');
- page.setSearchText('Country:FR');
- expect(page.getChangesAlertTableTitle('Alerts (169)')).toEqual('Alerts (25)');
- page.clickClearSearch();
-
- expect(page.getTableColumnNames()).toContain('Country', 'for renamed column names for alert list table');
-
- });
+ // it('should rename columns from table configuration', () => {
--- End diff --
Same here
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
@merrimanr That must be it, I didn't do a new pull. Now, to figure out how to do it with the checkout-pr script and not have to start over.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
Re-upping what I said before
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
Similar to https://issues.apache.org/jira/browse/METRON-876
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
@justinleet I think that is a great idea. I didn't do it initially to keep the size of this PR modest but I can add it in (we can always roll it back and move it to another PR if we want). I'll start with metron-elasticsearch and see how it goes.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr closed the pull request at:
https://github.com/apache/metron/pull/858
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/858
I think getting the feature branch set up was a great first step, so thanks for setting that up, @merrimanr.
I agree with @ottobackwards, that we need a discussion on what the next steps are. For me, I think it's getting reqs fleshed out and agreed on (i.e. what's the end state where we pull in the branch, and what's follow on). I think getting a clear idea of end state for this phase is going to help avoid feature creep and the branch being stagnant.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
I was more thinking about having to have a "Debugging Integration Tests" readme.
I think having the in-memory stuff available is worth getting some feedback.
I have in the past debugged through the kafka topic and storm stuff when looking to track down error messages and issues in the integration tests. I don't know if that is something that happens often enough to warrant keeping the in-memory stuff.
But, if we don't keep them, then would we want the docker testing containers to be setup for debugging? Would it work? So the start.sh starts with the debugging on, known ports etc.
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/858
Docker Compose provides a CLI just like vagrant does: https://docs.docker.com/compose/reference/overview/. I would be happy to wrap those commands in easy to understand scripts.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r156215449
--- Diff: metron-interface/metron-alerts/protractor.conf.js ---
@@ -25,27 +25,28 @@ var SpecReporter = require('jasmine-spec-reporter').SpecReporter;
exports.config = {
allScriptsTimeout: 25000,
specs: [
- './e2e/login/login.e2e-spec.ts',
- './e2e/alerts-list/alerts-list.e2e-spec.ts',
+ // './e2e/login/login.e2e-spec.ts',
--- End diff --
Done
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/858
I agree with @cestella (and this might spill over into a discuss thread as @ottobackwards mentioned). Maintaining both anything over other than short, short term is going to be a nightmare. Invariably one or the other is going to break independently of the other.
Assuming we do go with remote debugging of Docker (and that's a total assumption, to be clear), we should make sure that the instructions aren't IntelliJ dependent. Or at least, there are best effort instructions for people using something else. I.e. if we end up using something like JetBrains' Docker Integration plugin, I personally think we should have at least some debugging instructions available outside that.
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/858#discussion_r155339627
--- Diff: metron-interface/metron-alerts/e2e/utils/e2e_util.ts ---
@@ -46,25 +48,53 @@ export function waitForStalenessOf (_element ) {
}
export function loadTestData() {
--- End diff --
Agreed
---
[GitHub] metron pull request #858: METRON-1344: Externalize the infrastructural compo...
Posted by merrimanr <gi...@git.apache.org>.
GitHub user merrimanr reopened a pull request:
https://github.com/apache/metron/pull/858
METRON-1344: Externalize the infrastructural components using integration tests
## Contributor Comments
This PR will add infrastructure to our Travis build that will allow the Alerts UI e2e test to be run. There are several outstanding issues that still need to be worked out so DO NOT MERGE this yet.
This is a first pass at a potential Docker-based solution and is meant to be a POC. The intention is to facilitate further discussion around the general approach and provide a working example that we can build off of.
A good place to start is the .travis file. This provides a good guide on how this infrastructure is spun up. It is assumed Docker and Docker Compose are installed. To use outside of travis in a local dev environment (assuming Mac OSX):
1. Build Metron with `mvn clean install -DskipTests`
1. Create a Docker machine with `metron-contrib/metron-docker/scripts/create-docker-machine.sh`
1. Set the Docker env variables with `eval $(docker-machine env metron-machine)`
1. Build the base Metron image that installs Java: `docker build ./metron-centos/ -t "metron-centos"`
1. Spin up the environment: `cd metron-contrib/metron-docker-e2e/compose && docker-compose up -d`
A working environment should now be available. To verify get the Docker machine address with `echo $DOCKER_HOST`. Services should be available on the ports specified in `metron-contrib/metron-docker-e2e/compose/docker-compose.yml`. For example, assuming my $DOCKER_HOST is "tcp://192.168.99.100:2376", Elasticsearch should be available at http://192.168.99.100:9210.
At this point only a single e2e test is run due to the significant refactoring being done in https://github.com/apache/metron/pull/857.
Here are my thoughts so far based on work in this PR:
- The Docker environment creation and startup adds about 3 minutes to the build. This is about what I expected and hopefully we can get this time back as we move other tests to a reusable environment.
- I'm not convinced spinning up containers for the Alerts UI and REST are necessary or desired. We may be able to cut some time off the build by running them directly on the Travis host instead of in Docker
- I experimented with caching the Docker images but pulling them every time was actually faster
There is still significant work to be done including:
- I was able to get the full suite of tests to run successfully in previous commits but have all but e2e tests commented out for now to make it easier to see what's going on in Travis. These will need to be added back once we get closer to a final solution.
- Once the e2e test refactoring is done those changes will need merged in and tested
- The license check is commented out right now because I inadvertently added some new dependency versions (have no idea why). Still need to track that down.
- There is an intermittent error that happens when starting up REST. Still working on tracking this down.
Looking forward to some feedback.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron e2e-in-travis
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/858.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #858
----
commit b8fb307748c4e83e88cd9c156464eb99b5b13025
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:05:14Z
initial commit
commit faabe6789cb83abe6d5ba6005fa10f73225e253a
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:08:32Z
travis test
commit c61ca37491225df77c14363c499c7465bca15651
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:09:55Z
travis test
commit dda0d7a6d7264dbfcbfb8bdc6e6bf2fc3325ecdb
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:15:31Z
travis test
commit 083f4290b8a3129e27bcca15879bf3889361357a
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:19:20Z
travis test
commit 2b4a90e777756d24d1e74ec80cfbf74832f8d648
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T19:56:57Z
travis test
commit 7d9b47f5de63ceec66891eb4ee1e5122f908a2a9
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T20:18:32Z
travis test
commit 594d94c754ca6fc0e62a3fa82505598b0e71d7ae
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T20:38:49Z
travis test
commit cd06db23494f084abb05a3adbbb2b3170927aaf8
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:11:01Z
travis test
commit 75a8d13c4ecc07a69443aaa61d4ba4859f860946
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:17:22Z
travis test
commit cbf2da7597514a3c8e02309f463d8726d84c13be
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T21:49:07Z
travis test
commit 280fdf5bca481049ae0f83c5deae1f93611959fa
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:25:20Z
travis test
commit 0ef3f2a79824942edace28b57bf93975760177f6
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:31:03Z
travis test
commit b21ce8288bbbabd75e4eb2e877de20db81fe3b8d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:36:16Z
travis test
commit 4e741903281875f31ccee0c5d90868a329ff2a67
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:40:23Z
travis test
commit 2e6d74ac80ccc40f3f96e29640d5e3b07eba870d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:47:12Z
travis test
commit f3cb688563fc201ea4178a45369864184a176dec
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:51:06Z
travis test
commit 55ca067f3768f88f7dade0602a00af92f48e7616
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T22:56:04Z
travis test
commit a3c65db085744fc3b65491211a769cf19a0695e9
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:15:29Z
travis test
commit 4751848e7cf5221976e7bae23cea8fc9ee75a72f
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:18:16Z
travis test
commit 86a0a5db301b0246ae650d67e1e45c4a442ec074
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:27:23Z
travis test
commit ce2c75acbff2486f028543ef2a56ee3f21a58fc0
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:36:12Z
travis test
commit b5c5a94ef53bc154019c623d7a8368ff99950f9d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-01T23:48:56Z
travis test
commit ce2bf87199bfc5fb2db1c7d1df7c07ff992b8905
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T00:04:45Z
travis test
commit b426a28cc49e21db4907db5c35268c193e69a8da
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T00:49:24Z
travis test
commit 09aa8f080c6ccf3c4e9d5fb267fa86f113ee286c
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T01:22:07Z
travis test
commit 1e89a1dceea81c25411f596e29cb73964a408b17
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T01:23:36Z
travis test
commit 02c147bcc9543f3bc07c91e06dc11c9b35b3e422
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:15:21Z
travis test
commit a693cf95398e8300900530b085c4724b8a822026
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:21:58Z
travis test
commit 39d1d96b497b67b40e1aabed23c84a81ec3f009d
Author: merrimanr <me...@gmail.com>
Date: 2017-12-02T15:54:40Z
travis test
----
---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/858
Maybe a Jira for the feature branch, and each PR as a subtask?
---