You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/27 14:47:00 UTC
[jira] [Commented] (METRON-1644) Support parser chaining
[ https://issues.apache.org/jira/browse/METRON-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525162#comment-16525162 ]
ASF GitHub Bot commented on METRON-1644:
----------------------------------------
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/1084
METRON-1644: Support parser chaining
## Contributor Comments
Currently we have only one layer of parsing prior to enrichment, but often real data is more complex. For instance, often data is wrapped in an envelope (e.g. syslog data which contains a field which needs to be parsed).
This Jira allows us to support a DAG of parsers prior to enrichment by allowing us to provide a strategy for interpreting what is data and what is metadata in the parser bolt. This enables upstream parsers to pass in a JSON Blob which contains the metadata and have the parser bolt choose which field is the data to be parsed, the remaining fields would be considered metadata (and merged into the resulting data or not depending on our existing rules for handling metadata).
To illustrate this better, I've provided a use-case with an example. Note, this PR depends on METRON-1643 and METRON-1642, so those should be reviewed prior to this.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron enveloped_message_supplier
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1084.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1084
----
commit d1cb318961277da732650152c8526054769d830b
Author: cstella <ce...@...>
Date: 2018-06-05T21:07:13Z
Adding message supplier. Tests pending.
commit ffad876b377f6cf94213b783a8c139e0973c2e00
Author: cstella <ce...@...>
Date: 2018-06-06T14:12:55Z
Fixed tests
commit b349bf0b8fa251846d77cabc8b6aa019154a6425
Author: cstella <ce...@...>
Date: 2018-06-06T14:22:53Z
missing license
commit 71959c89816ddf4d84fef9585cfb03bd7f624612
Author: cstella <ce...@...>
Date: 2018-06-07T16:05:14Z
Renamed and refactored metadata utilities
commit 6e809e4cf9c34c9207d2d7c501db1dc0390c1d0b
Author: cstella <ce...@...>
Date: 2018-06-07T16:06:51Z
compilation issues
commit 5c9dd7caa68820d1ca06d99ff3ef2ac7eace7ba6
Author: cstella <ce...@...>
Date: 2018-06-07T20:55:10Z
Added unit and integration tests
commit bbd922e440e3dcfd22d54e76710d75d3fd2f5ab9
Author: cstella <ce...@...>
Date: 2018-06-07T22:10:00Z
Updating documentation and refactoring.
commit b2d1ca8b685306aafe07a65811f8223b6136bf9a
Author: cstella <ce...@...>
Date: 2018-06-08T02:36:49Z
Updated integration test.
commit 3b01987a64030a1dd3cccac2d70d7e5d375bac1c
Author: cstella <ce...@...>
Date: 2018-06-08T03:07:04Z
Merge branch 'master' into enveloped_message_supplier
commit f940234aec3e3360e12b4015c9cdc9cf01df681d
Author: cstella <ce...@...>
Date: 2018-06-08T03:30:07Z
Fixed test and subtle bug.
commit 92e92fa658d5aaeeb92b970449ac7dded0ed675e
Author: cstella <ce...@...>
Date: 2018-06-08T05:06:53Z
typos
commit 8c7112095ceddf5b7a1e812b905721991abd91ee
Author: cstella <ce...@...>
Date: 2018-06-08T05:11:02Z
typo
commit b9036f8f6feebbb19efb964a66f5ff8131fcbe8d
Author: cstella <ce...@...>
Date: 2018-06-08T05:21:28Z
missed sensorname
commit 532add3d533bd73ab7228a09e0dfc50255242c33
Author: cstella <ce...@...>
Date: 2018-06-08T06:21:44Z
updating grok
commit b5140a22fda7fd39f18b5a13c91a95785f3b6a4b
Author: cstella <ce...@...>
Date: 2018-06-08T06:33:38Z
Updating grok
commit cfcbcff3353a6773c18cebfa877e350a85e38859
Author: cstella <ce...@...>
Date: 2018-06-27T14:19:55Z
Merge branch 'master' into enveloped_message_supplier
----
> Support parser chaining
> -----------------------
>
> Key: METRON-1644
> URL: https://issues.apache.org/jira/browse/METRON-1644
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
> Priority: Major
>
> Currently we have only one layer of parsing prior to enrichment, but often real data is more complex. For instance, often data is wrapped in an envelope (e.g. syslog data which contains a field which needs to be parsed).
> This Jira allows us to support a DAG of parsers prior to enrichment by allowing us to provide a strategy for interpreting what is data and what is metadata in the parser bolt. This enables upstream parsers to pass in a JSON Blob which contains the metadata and have the parser bolt choose which field is the data to be parsed, the remaining fields would be considered metadata (and merged into the resulting data or not depending on our existing rules for handling metadata).
>
> To illustrate this better, I've provided a use-case with an example. Note, this PR depends on METRON-1643 and METRON-1642, so those should be reviewed prior to this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)