You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by cestella <gi...@git.apache.org> on 2018/06/27 14:46:37 UTC

[GitHub] metron pull request #1084: METRON-1644: Support parser chaining

GitHub user cestella opened a pull request:

    https://github.com/apache/metron/pull/1084

    METRON-1644: Support parser chaining

    ## Contributor Comments
    Currently we have only one layer of parsing prior to enrichment, but often real data is more complex.  For instance, often data is wrapped in an envelope (e.g. syslog data which contains a field which needs to be parsed).
    
    This Jira allows us to support a DAG of parsers prior to enrichment by allowing us to provide a strategy for interpreting what is data and what is metadata in the parser bolt.  This enables upstream parsers to pass in a JSON Blob which contains the metadata and have the parser bolt choose which field is the data to be parsed, the remaining fields would be considered metadata (and merged into the resulting data or not depending on our existing rules for handling metadata).
    
    To illustrate this better, I've provided a use-case with an example.  Note, this PR depends on METRON-1643 and METRON-1642, so those should be reviewed prior to this.
    
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron.  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
      ```
      mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
      ```
    
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron enveloped_message_supplier

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1084.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1084
    
----
commit d1cb318961277da732650152c8526054769d830b
Author: cstella <ce...@...>
Date:   2018-06-05T21:07:13Z

    Adding message supplier.  Tests pending.

commit ffad876b377f6cf94213b783a8c139e0973c2e00
Author: cstella <ce...@...>
Date:   2018-06-06T14:12:55Z

    Fixed tests

commit b349bf0b8fa251846d77cabc8b6aa019154a6425
Author: cstella <ce...@...>
Date:   2018-06-06T14:22:53Z

    missing license

commit 71959c89816ddf4d84fef9585cfb03bd7f624612
Author: cstella <ce...@...>
Date:   2018-06-07T16:05:14Z

    Renamed and refactored metadata utilities

commit 6e809e4cf9c34c9207d2d7c501db1dc0390c1d0b
Author: cstella <ce...@...>
Date:   2018-06-07T16:06:51Z

    compilation issues

commit 5c9dd7caa68820d1ca06d99ff3ef2ac7eace7ba6
Author: cstella <ce...@...>
Date:   2018-06-07T20:55:10Z

    Added unit and integration tests

commit bbd922e440e3dcfd22d54e76710d75d3fd2f5ab9
Author: cstella <ce...@...>
Date:   2018-06-07T22:10:00Z

    Updating documentation and refactoring.

commit b2d1ca8b685306aafe07a65811f8223b6136bf9a
Author: cstella <ce...@...>
Date:   2018-06-08T02:36:49Z

    Updated integration test.

commit 3b01987a64030a1dd3cccac2d70d7e5d375bac1c
Author: cstella <ce...@...>
Date:   2018-06-08T03:07:04Z

    Merge branch 'master' into enveloped_message_supplier

commit f940234aec3e3360e12b4015c9cdc9cf01df681d
Author: cstella <ce...@...>
Date:   2018-06-08T03:30:07Z

    Fixed test and subtle bug.

commit 92e92fa658d5aaeeb92b970449ac7dded0ed675e
Author: cstella <ce...@...>
Date:   2018-06-08T05:06:53Z

    typos

commit 8c7112095ceddf5b7a1e812b905721991abd91ee
Author: cstella <ce...@...>
Date:   2018-06-08T05:11:02Z

    typo

commit b9036f8f6feebbb19efb964a66f5ff8131fcbe8d
Author: cstella <ce...@...>
Date:   2018-06-08T05:21:28Z

    missed sensorname

commit 532add3d533bd73ab7228a09e0dfc50255242c33
Author: cstella <ce...@...>
Date:   2018-06-08T06:21:44Z

    updating grok

commit b5140a22fda7fd39f18b5a13c91a95785f3b6a4b
Author: cstella <ce...@...>
Date:   2018-06-08T06:33:38Z

    Updating grok

commit cfcbcff3353a6773c18cebfa877e350a85e38859
Author: cstella <ce...@...>
Date:   2018-06-27T14:19:55Z

    Merge branch 'master' into enveloped_message_supplier

----


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    Ok, more javadocs is definitely fair.  I went through the core abstractions and added javadocs.  If I missed anything, let me know.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    1 nit, other than that +1


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    Ok, the prerequisite tickets are merged here and this is ready for review.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    This is significant enough that I think some level of design write-up is warranted.  At some point we'll want to update the top level doc's and diagrams, but I'm OK with that being a follow on.  But some description of the approach and how it works is required for proper review.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    @ottobackwards Yeah, that is a really good point.  I'll write up something.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    @ottobackwards Ok, I attempted to do that [here](https://github.com/cestella/incubator-metron/blob/c4e4786e778d5b06cd16f7faa7d3522f620fc2ba/metron-platform/metron-parsers/ParserChaining.md).  Can you help me understand where the gaps are.  Is it one or more of the following:
    [ ] The document doesn't have enough detail about what was done before, just detail about the current architecture.
    [ ] The document does not link to relevant code examples to create custom raw message strategies.
    [ ] The document is too high level
    [ ] The document is too low level
    [ ] Other.  Please specify details if this is checked.


---

[GitHub] metron pull request #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1084#discussion_r201448152
  
    --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/message/metadata/EnvelopedRawMessageStrategy.java ---
    @@ -0,0 +1,146 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.metron.common.message.metadata;
    +
    +import org.apache.metron.common.Constants;
    +import org.apache.metron.common.utils.JSONUtils;
    +import org.json.simple.JSONObject;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.lang.invoke.MethodHandles;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +/**
    + * An alternative strategy whereby
    + * <ul>
    + *  <li>The raw data is presumed to be a JSON Map</li>
    + *  <li>The data to be parsed is the contents of one of the fields.</li>
    + *  <li>The non-data fields are considered metadata</li>
    + * </ul>
    + *
    + * Additionally, the defaults around merging and reading metadata are adjusted to be on by default.
    + * Note, this strategy allows for parser chaining and for a fully worked example, check the parser chaining use-case.
    + */
    +public class EnvelopedRawMessageStrategy implements RawMessageStrategy {
    +  private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
    +  /**
    +   * The field from the rawMessageStrategyConfig in the SensorParserConfig that defines the field to use to
    +   * define the data to be parsed.
    +   */
    +  public static final String MESSAGE_FIELD_CONFIG = "messageField";
    +
    +  /**
    +   * Retrieve the raw message by parsing the JSON Map in the kafka value and pulling the appropriate field.
    +   * Also, augment the default metadata with the non-data fields in the JSON Map.
    +   *
    +   * Note: The data field in the JSON Map is not considered metadata.
    +   *
    +   * @param rawMetadata The metadata read from kafka Key (e.g. the topic, index, etc.)
    +   * @param rawMessage The raw message from the kafka value
    +   * @param readMetadata True if we want to read read the metadata
    +   * @param config The config for the RawMessageStrategy (See the rawMessageStrategyConfig in the SensorParserConfig)
    +   * @return
    +   */
    +  @Override
    +  public RawMessage get(Map<String, Object> rawMetadata, byte[] rawMessage, boolean readMetadata, Map<String, Object> config) {
    +    String messageField = (String)config.get(MESSAGE_FIELD_CONFIG);
    +    if(messageField == null) {
    +      throw new IllegalStateException("You must specify a message field in the message supplier config.  " +
    +              "I expected to find a \"messageField\" field in the config.");
    --- End diff --
    
    haha


---

[GitHub] metron pull request #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:

    https://github.com/apache/metron/pull/1084


---

[GitHub] metron pull request #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:

    https://github.com/apache/metron/pull/1084

    METRON-1644: Support parser chaining

    ## Contributor Comments
    Currently we have only one layer of parsing prior to enrichment, but often real data is more complex.  For instance, often data is wrapped in an envelope (e.g. syslog data which contains a field which needs to be parsed).
    
    This Jira allows us to support a DAG of parsers prior to enrichment by allowing us to provide a strategy for interpreting what is data and what is metadata in the parser bolt.  This enables upstream parsers to pass in a JSON Blob which contains the metadata and have the parser bolt choose which field is the data to be parsed, the remaining fields would be considered metadata (and merged into the resulting data or not depending on our existing rules for handling metadata).
    
    To illustrate this better, I've provided a use-case with an example.  Note, this PR depends on METRON-1643 and METRON-1642, so those should be reviewed prior to this.
    
    Manual Test script:
    * Go through the use-case doc.
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron.  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    
    ### For code changes:
    - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [x] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
      ```
      mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
      ```
    
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
    It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron enveloped_message_supplier

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1084.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1084
    
----
commit 5030d00f9eb13ff662716f1c05a731938a66cb52
Author: cstella <ce...@...>
Date:   2018-06-05T19:05:59Z

    Regex based message router transformer.

commit d1cb318961277da732650152c8526054769d830b
Author: cstella <ce...@...>
Date:   2018-06-05T21:07:13Z

    Adding message supplier.  Tests pending.

commit ffad876b377f6cf94213b783a8c139e0973c2e00
Author: cstella <ce...@...>
Date:   2018-06-06T14:12:55Z

    Fixed tests

commit b349bf0b8fa251846d77cabc8b6aa019154a6425
Author: cstella <ce...@...>
Date:   2018-06-06T14:22:53Z

    missing license

commit 7c2723bc243996a2a0606e07def5e3b5c1736e59
Author: cstella <ce...@...>
Date:   2018-06-06T14:35:25Z

    updating test

commit 71959c89816ddf4d84fef9585cfb03bd7f624612
Author: cstella <ce...@...>
Date:   2018-06-07T16:05:14Z

    Renamed and refactored metadata utilities

commit 6e809e4cf9c34c9207d2d7c501db1dc0390c1d0b
Author: cstella <ce...@...>
Date:   2018-06-07T16:06:51Z

    compilation issues

commit 5c9dd7caa68820d1ca06d99ff3ef2ac7eace7ba6
Author: cstella <ce...@...>
Date:   2018-06-07T20:55:10Z

    Added unit and integration tests

commit bbd922e440e3dcfd22d54e76710d75d3fd2f5ab9
Author: cstella <ce...@...>
Date:   2018-06-07T22:10:00Z

    Updating documentation and refactoring.

commit b2d1ca8b685306aafe07a65811f8223b6136bf9a
Author: cstella <ce...@...>
Date:   2018-06-08T02:36:49Z

    Updated integration test.

commit 4a5461a8a7b4cd34d6541c0449470901d6ef3ed5
Author: cstella <ce...@...>
Date:   2018-06-08T02:56:50Z

    Merge branch 'master' into regex_routing

commit f1bda7478824e716d60a28cab8680e303a9e28a7
Author: cstella <ce...@...>
Date:   2018-06-08T03:03:34Z

    regex routing docs

commit 3b01987a64030a1dd3cccac2d70d7e5d375bac1c
Author: cstella <ce...@...>
Date:   2018-06-08T03:07:04Z

    Merge branch 'master' into enveloped_message_supplier

commit f940234aec3e3360e12b4015c9cdc9cf01df681d
Author: cstella <ce...@...>
Date:   2018-06-08T03:30:07Z

    Fixed test and subtle bug.

commit 92e92fa658d5aaeeb92b970449ac7dded0ed675e
Author: cstella <ce...@...>
Date:   2018-06-08T05:06:53Z

    typos

commit 8c7112095ceddf5b7a1e812b905721991abd91ee
Author: cstella <ce...@...>
Date:   2018-06-08T05:11:02Z

    typo

commit b9036f8f6feebbb19efb964a66f5ff8131fcbe8d
Author: cstella <ce...@...>
Date:   2018-06-08T05:21:28Z

    missed sensorname

commit 532add3d533bd73ab7228a09e0dfc50255242c33
Author: cstella <ce...@...>
Date:   2018-06-08T06:21:44Z

    updating grok

commit b5140a22fda7fd39f18b5a13c91a95785f3b6a4b
Author: cstella <ce...@...>
Date:   2018-06-08T06:33:38Z

    Updating grok

commit ba862789e43f9e5d8e648d86378d89ec98d30d5b
Author: cstella <ce...@...>
Date:   2018-06-27T14:19:42Z

    Merge branch 'master' into regex_routing

commit cfcbcff3353a6773c18cebfa877e350a85e38859
Author: cstella <ce...@...>
Date:   2018-06-27T14:19:55Z

    Merge branch 'master' into enveloped_message_supplier

commit e807b0d7b61ac8b77b24051c8e0eedb53aa9be9d
Author: cstella <ce...@...>
Date:   2018-06-27T15:11:25Z

    Phrasing! we ARE still doing phrasing, right?

commit 8a96773c72077785eff538abbf359587b71d8e28
Author: cstella <ce...@...>
Date:   2018-06-27T15:22:39Z

    Renaming to REGEX_SELECT

commit 28cbfaff4a24de43c4752761dfbfa9e7d5577274
Author: cstella <ce...@...>
Date:   2018-06-27T15:22:53Z

    Merge branch 'regex_routing' into enveloped_message_supplier

commit 9218b469367a917b0fb004a473f06188483a10be
Author: cstella <ce...@...>
Date:   2018-06-27T15:23:29Z

    Updating naming change

commit 2cd91268c0a8de9d86b01e5613828f6ded8deb9b
Author: cstella <ce...@...>
Date:   2018-06-27T21:26:51Z

    Creating a parser chaining document

commit c4e4786e778d5b06cd16f7faa7d3522f620fc2ba
Author: cstella <ce...@...>
Date:   2018-07-06T15:44:41Z

    Merge branch 'master' into enveloped_message_supplier

commit 2623660f9191a7596c53b623c031e13b75e92fd7
Author: cstella <ce...@...>
Date:   2018-07-06T18:35:35Z

    Updating with javadocs

commit efbe06e245506a9a25c0fd1cf5a3d173fb8f1f81
Author: cstella <ce...@...>
Date:   2018-07-06T18:50:53Z

    Fixed readme

----


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    +1, once @ottobackwards is good with everything.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    I'm looking for a doc that describes how these new things work, like what would have come out of a discuss thread if there had been discussion on this before hand.
    
    " Dev. design notes"
    "The way this works is ...   We use the configuration ago do this, and the new strategies to do that, basically setting the system up so that we can do x,y and z in this component.  See this code, that code and this code."  "Other ways to do this are foo and bar, but I chose this because cause blah".



---

[GitHub] metron pull request #1084: METRON-1644: Support parser chaining

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/metron/pull/1084


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    Let's try something else.  Please javadoc all the new classes and functionality, such that someone else if they want to review or maintain this can understand their implementation


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    I was able to spin this up and run through the README example, with the fix from the most recent commit.  Looks like something's going on in Travis, unfortunately.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    Yes.  The use case is useful, but this is more dev. focused, if that makes sense.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    @ottobackwards Ok, let me know what you think about [this](https://github.com/cestella/incubator-metron/blob/enveloped_message_supplier/metron-platform/metron-parsers/ParserChaining.md).  Is it too dev focused?  I tried to lay out the high level knobs to enable the use-case and the justifications for them, but I might've failed at it.


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining (NOTE: review METRO...

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    Yeah, I tend to agree @ottobackwards .  There is a high level diagram in the use-case under the "High Level Solution" section; do you think that should be extracted out into its own document along with some discussion of the strategy pattern used for extracting metadata and data?


---

[GitHub] metron pull request #1084: METRON-1644: Support parser chaining

Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1084#discussion_r201426118
  
    --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/message/metadata/EnvelopedRawMessageStrategy.java ---
    @@ -0,0 +1,146 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.metron.common.message.metadata;
    +
    +import org.apache.metron.common.Constants;
    +import org.apache.metron.common.utils.JSONUtils;
    +import org.json.simple.JSONObject;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.io.IOException;
    +import java.lang.invoke.MethodHandles;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +/**
    + * An alternative strategy whereby
    + * <ul>
    + *  <li>The raw data is presumed to be a JSON Map</li>
    + *  <li>The data to be parsed is the contents of one of the fields.</li>
    + *  <li>The non-data fields are considered metadata</li>
    + * </ul>
    + *
    + * Additionally, the defaults around merging and reading metadata are adjusted to be on by default.
    + * Note, this strategy allows for parser chaining and for a fully worked example, check the parser chaining use-case.
    + */
    +public class EnvelopedRawMessageStrategy implements RawMessageStrategy {
    +  private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
    +  /**
    +   * The field from the rawMessageStrategyConfig in the SensorParserConfig that defines the field to use to
    +   * define the data to be parsed.
    +   */
    +  public static final String MESSAGE_FIELD_CONFIG = "messageField";
    +
    +  /**
    +   * Retrieve the raw message by parsing the JSON Map in the kafka value and pulling the appropriate field.
    +   * Also, augment the default metadata with the non-data fields in the JSON Map.
    +   *
    +   * Note: The data field in the JSON Map is not considered metadata.
    +   *
    +   * @param rawMetadata The metadata read from kafka Key (e.g. the topic, index, etc.)
    +   * @param rawMessage The raw message from the kafka value
    +   * @param readMetadata True if we want to read read the metadata
    +   * @param config The config for the RawMessageStrategy (See the rawMessageStrategyConfig in the SensorParserConfig)
    +   * @return
    +   */
    +  @Override
    +  public RawMessage get(Map<String, Object> rawMetadata, byte[] rawMessage, boolean readMetadata, Map<String, Object> config) {
    +    String messageField = (String)config.get(MESSAGE_FIELD_CONFIG);
    +    if(messageField == null) {
    +      throw new IllegalStateException("You must specify a message field in the message supplier config.  " +
    +              "I expected to find a \"messageField\" field in the config.");
    --- End diff --
    
    Not sure about the first person here.  I'm all for the @cestella / metron convergence, but let us not rush into that good night


---

[GitHub] metron issue #1084: METRON-1644: Support parser chaining

Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:

    https://github.com/apache/metron/pull/1084
  
    Intermittent test failure, travis sorted now.



---