You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/30 13:14:00 UTC
[jira] [Commented] (METRON-1815) Separate metron-parsers into
metron-parsers-common and metron-parsers-storm
[ https://issues.apache.org/jira/browse/METRON-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668681#comment-16668681 ]
ASF GitHub Bot commented on METRON-1815:
----------------------------------------
GitHub user justinleet opened a pull request:
https://github.com/apache/metron/pull/1249
METRON-1815: Separate metron-parsers into metron-parsers-common and metron-parsers-storm
## Contributor Comments
Separating metron-parsers into metron-parsers-common and metron-parsers-storm in order to decouple the parsers from Storm. This builds on the work from [METRON-1681](https://issues.apache.org/jira/browse/METRON-1681), done here: https://github.com/apache/metron/pull/1213. It looks like a ton of changes, but most of it is pretty minor naming adjustments, moving whole files, etc.
Full dev should spin up and run exactly as it does prior to this PR, which is also how to test this.
At this point, no org.apache.storm dependency should remain in metron-parsers-common (as that now lives in metron-parsers-storm). This means that we are typically going to be running, for now, off of metron-parsers-storm jars.
Nothing changes in Ambari, because there's no actual second platform we're running on. In the future, I'd say we need to make some sort of configuration around this, but I'm fine with that being follow-on whenever we choose to build off of something other than Storm.
This also adjusts some of the integration tests, so that it's easier to extend to a new platform once we add something beyond Storm. The current solution would involve a class for every parser for every platform, which seems awful. The new solution uses a parameterized test, and simply one extended class that's platform specific. Let me know thoughts on this and if we want to refactor this differently.
The commit history is a little wonky, since this started based on his PR before it made it into master and there was a variety of merging happening there. The commits from @merrimanr are from that branch, not coauthored here. I'm a bit surprised git didn't handle it properly, but it won't matter once squashed.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justinleet/metron parserSplit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1249.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1249
----
commit 922c76316e5fa2d22976c938f4d4db8c45ec6329
Author: merrimanr <me...@...>
Date: 2018-09-26T16:04:57Z
initial commit
commit 773c34a30f29e7353ecd5fd8bf5fa0f545219eec
Author: merrimanr <me...@...>
Date: 2018-09-26T18:00:11Z
removed commented code
commit 745dd4fced3c13e8538ea974afa1b95dfd08e214
Author: justinjleet <ju...@...>
Date: 2018-10-03T15:41:37Z
wip
commit c0e52242d01db70f580e26d2e7af7a0bf5c33468
Author: justinjleet <ju...@...>
Date: 2018-10-03T20:01:23Z
fix
commit 17b3ab1e47fdb0ece8550c1d9f112acc777e45bd
Author: justinjleet <ju...@...>
Date: 2018-10-03T20:01:26Z
Merge branch 'master' into parserSplit
commit 2bbd9962251855158a26c55c45b2b4fbf1d9f9e8
Author: merrimanr <me...@...>
Date: 2018-10-03T21:15:32Z
feedback from nick
commit 52ec5d9a58bc390527eef61964a2fd59e86057ea
Author: justinjleet <ju...@...>
Date: 2018-10-03T21:22:26Z
pom fixes
commit 5da67a3acc51a757a5d5923ab37dd748d3b008de
Author: justinjleet <ju...@...>
Date: 2018-10-04T19:49:04Z
WIP, working full dev
commit 2650b9090d41bc9563a2d8e9ed147eefbf3b7591
Author: merrimanr <me...@...>
Date: 2018-10-04T20:35:56Z
removed callbacks
commit 92c11b79988647aae01e2abe326df6301ca9e2d7
Author: justinjleet <ju...@...>
Date: 2018-10-04T20:45:18Z
fixing some docs + pom versioning that was hitting cache
commit 599133dc71d742f928def4eef0dba2121d9a1666
Author: merrimanr <me...@...>
Date: 2018-10-04T21:59:25Z
Merge remote-tracking branch 'mirror/master' into METRON-1681
commit 901f5958fcbf8582618c38634edae034530cf0d4
Author: justinjleet <ju...@...>
Date: 2018-10-05T15:37:55Z
renaming module
commit c517acb1aad65fce89fdff23f6309801bbea55df
Author: justinjleet <ju...@...>
Date: 2018-10-05T15:38:52Z
Merge branch 'master' into parserSplit
commit f473110a4e9814739eb2bd369b0c529e47ddd882
Author: justinjleet <ju...@...>
Date: 2018-10-05T18:05:36Z
fix stellar jar path
commit de509605ae6315433bf2396949e941db4db3a0d7
Author: justinjleet <ju...@...>
Date: 2018-10-05T21:07:17Z
Merge branch 'METRON-1681' into parserSplit
commit 1aab7277d7988b2f311756ce5a77c21b686a19d1
Author: justinjleet <ju...@...>
Date: 2018-10-09T14:02:48Z
Updating testing readme, removing comments
commit 95b61008ec5e3c6a7628e06947d2d8168bd2765d
Author: merrimanr <me...@...>
Date: 2018-10-09T22:38:10Z
added hashcode method
commit e21037fa0c6748d965efe493ad0a60d17fef1e07
Author: justinjleet <ju...@...>
Date: 2018-10-10T13:02:51Z
Merge branch 'METRON-1681' into parserSplit
commit b75e1a39b25361b8c18f73f8326184ebef1d7885
Author: merrimanr <me...@...>
Date: 2018-10-10T17:28:25Z
Merge remote-tracking branch 'mirror/master' into METRON-1681
# Conflicts:
# metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
# metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java
commit 9f1ce3e479d52f2142efaa9f527e2256ef1a7b38
Author: merrimanr <me...@...>
Date: 2018-10-11T23:07:51Z
resolved conflicts with METRON-1761
commit 7514549d8017ffb12496f2aaea147fe0f6638f02
Author: justinjleet <ju...@...>
Date: 2018-10-12T13:25:01Z
Merge branch 'METRON-1681' into parserSplit
commit 9365d6060387576594d6531b6cfe22326a7dbfad
Author: merrimanr <me...@...>
Date: 2018-10-15T20:53:26Z
Merge remote-tracking branch 'mirror/master' into METRON-1681
# Conflicts:
# metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
# metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java
commit 40ff3dd17f3dc0793089205e1113c319962e4d04
Author: merrimanr <me...@...>
Date: 2018-10-16T18:27:16Z
merged in master
commit 63722902bd091f75ecd8523958ae21460967ce90
Author: merrimanr <me...@...>
Date: 2018-10-17T15:34:48Z
deprecated methods
commit a31284402fb021661e1923986a9fd520bf7c3447
Author: justinjleet <ju...@...>
Date: 2018-10-17T15:48:50Z
Merge branch 'METRON-1681' into parserSplit
commit ea197f0bf3cb94aeed8d31dc6a252dbd786cb6ec
Author: justinjleet <ju...@...>
Date: 2018-10-18T14:01:59Z
READMEs rearranged
commit a432046f81708569ee87d7058a73ce33c1e67a96
Author: justinjleet <ju...@...>
Date: 2018-10-30T02:11:41Z
Merge branch 'master' into parserSplit
commit b392a5bbb11415905c6ca4fb4432183902beac27
Author: justinjleet <ju...@...>
Date: 2018-10-30T03:07:24Z
typo
----
> Separate metron-parsers into metron-parsers-common and metron-parsers-storm
> ---------------------------------------------------------------------------
>
> Key: METRON-1815
> URL: https://issues.apache.org/jira/browse/METRON-1815
> Project: Metron
> Issue Type: Improvement
> Reporter: Justin Leet
> Assignee: Justin Leet
> Priority: Major
>
> In order to expose our parsers to 3rd party components (e.g. the discussions on NiFi and potentially other platforms like Spark), we should
> separate the storm-bits into its own project. The metron-parsers-common
> project should contain only parser-oriented code, whereas the
> metron-parsers-storm project should contain the storm specific code
> (e.g. the parser bolt).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)