You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by merrimanr <gi...@git.apache.org> on 2016/05/12 22:13:44 UTC
[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...
GitHub user merrimanr opened a pull request:
https://github.com/apache/incubator-metron/pull/120
Metron 142 Simplify Parser configuration
This pull request includes several changes related to optimizing configuration for parser topologies. Currently the process for adding parsers includes several unnecessary steps with configuration being duplicated in various places. The high level changes are:
- Sensor specific config files (parser and enrichment) were reorganized into their respective projects
- Parser topology configuration is now dynamically driven by simple sensor specific json config files
- Zookeeper configuration infrastructure was extended to support parser topologies
- Ansible scripts were updated to load parser configurations to zookeeper and call parser topology startup scripts correctly.
- Unit and integration tests were updated to accommodate parser topology changes and provide coverage for new classes.
- Updated tools for reading and writing sensor parser configs to zookeeper
- Removed generic configuration feature as it is not being used anywhere
- Changed enrichment integration and unit tests to use a dedicated test config instead of reusing a sensor specific enrichment config
- Moved zookeeper configuration functions to a new component in the integration testing framework.
- Abstracted integration test validations to facilitate more readable code and reusable validations
The major benefits of this pull request are a simpler process for adding more parsers and additional test coverage for sensor specific parser configurations. This has been integration tested and verified with a full-dev-platform build.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-142
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/120.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #120
----
commit a8b2ece52d60ba7c69685455b38d3e71ec052979
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-02T14:39:33Z
moved parser topology from flux to java
commit d8317b14176278be3ef1bba4b8bb5b5423b83b57
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-03T22:31:58Z
finished integration test
commit 3b20cf7879faae9791856da08dc4dde25bad1a71
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-05T15:51:13Z
Merge remote-tracking branch 'apache/master' into parser-refactor
Conflicts:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
commit 136f76a539fc2378a63086f36c09271c950ccc3d
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-06T18:19:58Z
Merge remote-tracking branch 'apache/master' into parser-refactor
commit 7e511d49c50f588207bf16af26450e33b76c5152
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-06T22:36:25Z
METRON-142: Simplify Parser configuration
commit 5a05dcbed7bb235fbc0feedd05b0da26d7ce6ca6
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-06T22:43:47Z
METRON-142: Simplify Parser configuration
commit 5bebc79fea5344fe3799431a52114b4c4672c61b
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-06T23:25:52Z
Removed old parser integration tests
commit 79d02def004f12f6df4387ad9b5a26963bc46916
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-07T17:11:58Z
added mvn clean to travis build
commit de4292c6d1fed4d86ad1c29c24d322edf7b562cb
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-09T13:43:47Z
Removed uneeded failing pcap service test
commit 109b543f3cc04cd5a4066f85af9c47e874eb36a6
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-11T13:18:45Z
Merge remote-tracking branch 'apache/master' into METRON-142
Conflicts:
metron-deployment/roles/metron_streaming/files/config/sensors/pcap.json
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
metron-platform/metron-common/src/test/java/org/apache/metron/common/bolt/ConfiguredEnrichmentBoltTest.java
metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBolt.java
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/utils/SampleUtil.java
commit c8a142ec670316580690101705e136d914942169
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-12T15:43:53Z
Resolve merge conflicts
commit e33119529c0c2b4843ba9f3c77f13bd43a93b370
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-12T16:38:59Z
Sensor Parser integration tests now implemented in separate classes
commit 44494198827ec1d92dfeabe8697ceffe22b1e773
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-12T20:40:06Z
Merge remote-tracking branch 'apache/master' into METRON-142
commit 512b7e7ac9db0238a7a22bd1c4a42de534a969e7
Author: rmerriman <rm...@hortonworks.com>
Date: 2016-05-12T20:46:22Z
Removed old pcap.json file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/120#discussion_r63255049
--- Diff: metron-deployment/roles/metron_streaming/tasks/metron_topology.yml ---
@@ -16,7 +16,7 @@
#
---
- name: Submit Metron Parser topologies
--- End diff --
Based on recent changes, we no longer submit the topologies as part of the deployment process. The topologies (and most other services) are started at the end of the deployment only if the user has configured them to start.
Based on the new arguments, we'll need updates in the monit space. I'm thinking at least these will be impacted.
- `metron-deployment/roles/monit/templates/monit/parsers.monit`
- `metron-deployment/roles/monit/templates/scripts/start_topology.sh`
Let me know if you'd like help tackling that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/120#issuecomment-219493566
+1 Looks good. Validated 'Quick Dev' based on changes to how Monit is integrated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/120
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/120#issuecomment-219424532
This looks good to me. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---