[jira] [Commented] (METRON-1397) JSONMap parser should support JSON Path expressions to split input into multiple messages

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343968#comment-16343968 ] 

ASF GitHub Bot commented on METRON-1397:
----------------------------------------

GitHub user ottobackwards opened a pull request:

    https://github.com/apache/metron/pull/914

    METRON-1397  Support for JSON Path and complex documents in JSONMapParser

    It would be useful for implementors to have the ability to ingest more complex documents without having to stand up a NiFi node/cluster.
    
    This PR adds support for splitting multiple messages from a single message by way of using JSON Path statements to select the List element in the document.
    
    Since the parse() interface already returns a List<JSONObject> the scope of this change is in the parser.
    
    Example:
    
    ```json
        {
        "foo" :
        [
        { "name" : "foo1", "value" : "bar", "number" : 1.0 },
        { "name" : "foo2", "value" : "baz", "number" : 2.0 }
        ]
        }
     ```
    
    with JSON Path statement "$.foo" will result in two messages emitted from the parser
    
    ```json
     { "name" : "foo1", "value" : "bar", "number" : 1.0 }
    ```
    
    ```json
    { "name" : "foo2", "value" : "baz", "number" : 2.0 }
    ```
    Support in the configuration for a new configuration parameter: jsonpQuery has been added.
    An example of the new configuration:
    
    ```json
    {
      "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
      "sensorTopic":"jsonMapQuery",
      "parserConfig": {"jsonpQuery":"$.foo"}
    }
    ```
    
    In order to integration test both with and without the query present, a new jsonMapQuery parser configuration has been added, with test data.
    
    
    ##Testing
    - tests should run
    - deployment should work
    
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    
    ### For code changes:
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested manually?
    - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
    
    
    - [x] Have you written or updated unit tests and or integration tests to verify your changes?
    - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ottobackwards/metron jsonp-support

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #914
    
----
commit 414611b9ea6eb06b79cba0cc86e63ea062335884
Author: Otto Fowler <ot...@...>
Date:   2018-01-29T16:58:16Z

    JSON Path support for JSONMapParser
    Tests, Integration Tests, Doc

commit ad03ac4d3e7c98fcf571f3f3906efbb915deb1f9
Author: Otto Fowler <ot...@...>
Date:   2018-01-29T17:13:10Z

    checkstyle fixes

commit 7c76c5f5b7507ac55861ed44ad131700c1e6f124
Author: Otto Fowler <ot...@...>
Date:   2018-01-29T19:36:16Z

    account for new config in rpm
    
    fix handle empty

----


> JSONMap parser should support JSON Path expressions to split input into multiple messages
> -----------------------------------------------------------------------------------------
>
>                 Key: METRON-1397
>                 URL: https://issues.apache.org/jira/browse/METRON-1397
>             Project: Metron
>          Issue Type: New Feature
>            Reporter: Otto Fowler
>            Assignee: Otto Fowler
>            Priority: Major
>
> The parser interface takes byte[] and returns a list<jsonobject>.  
> It is consistent with the intent of the interface that any parser may return multiple messages from a given byte input.
> For the json map parser, we can make it easier for users (esp. if they do not have NiFi ) is we support more intake options out of the box, for instance more complex json documents containing lists of items.
> The parser should support a configuration option that declares a jsonpath statement, which will be evaluated to yield one or more 'sub' objects to be evaluated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)