You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by ottobackwards <gi...@git.apache.org> on 2016/09/24 13:58:43 UTC
[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...
GitHub user ottobackwards opened a pull request:
https://github.com/apache/incubator-metron/pull/271
METRON-449 JSONMapParser should unfold maps to arbitrary depths
The JSON Parser currently only unfolds maps at the root level. The parser would support a wider range of sources is it supported unfolding maps at any depth of the document.
METRON-449 adds a recursive function to unfold Maps as they are encountered when parsing. A new function was added because the enum lambdas cannot reference themselves.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ottobackwards/incubator-metron METRON-449
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #271
----
commit b5360fff99620d8b6806e1e0fa382b292fa56b8c
Author: Otto Fowler <of...@industrialdefender.com>
Date: 2016-09-24T13:53:31Z
METRON-449 JSONMapParser should unfold maps to arbitrary depths
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/271
@cestella , from looking at the ES templates etc, it seems that ':' is used as the name separator and not '.'. Should the the unfolding not use the same?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/271#discussion_r80380958
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/json/JSONMapParserTest.java ---
@@ -56,7 +56,7 @@ public void testHappyPath() {
/**
{
- "collection" : { "blah" : 7, "blah2" : "foo" }
+ "collection" : { "blah" : 7, "blah2" : "foo", "bigblah" : { "innerBlah" : "baz", "reallyInnerBlah" : { "color" : "grey" }}}
--- End diff --
Sure, I'll do that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/271
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/271
+1, tested in single node vagrant.
Did the following:
* Created a config called `custom` at `$METRON_HOME/config/zookeeper/parsers/custom.json`:
```
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"custom",
"parserConfig" : {
"mapStrategy" : "UNFOLD"
}
}
```
* Created kafka topic for `custom` via `/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic custom --partitions 1 --replication-factor 1`
* Loaded the configs into zookeeper via `$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i /usr/metron/0.2.0BETA/config/zookeeper -z node1:2181`
* Started the topology via `/usr/metron/0.2.0BETA/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s custom`
* Created some sample data in `~/test_data.dat`:
```
{ "foo" : "bar" }
{ "map" : { "blah" : 1 } }
{ "foo" : "grok", "map" : { "blah" : { "tim" : "sort" } } }
```
* Loaded the sample data into the kafka queue via `cat ~/test_data.dat | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic custom`
* Checked in ES Head that I had a message with a field called `map:blah:tim`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/271
No, ES uses ':' and modifies the keys upon write converting '.' to ':'.
Before that, '.' is the preferred source. I'm hoping we can go back to '.'
eventually. Also, non-ES indices are perfectly fine with '.'.
On Mon, Sep 26, 2016 at 11:59 AM, ottobackwards <no...@github.com>
wrote:
> @cestella <https://github.com/cestella> , from looking at the ES
> templates etc, it seems that ':' is used as the name separator and not '.'.
> Should the the unfolding not use the same?
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-metron/pull/271#issuecomment-249613819>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAg-x-luOvx1Ebrf2a3eH3JTi9nEd5rrks5qt-v4gaJpZM4KFphg>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/271
Never mind - just found the field name converter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/271#discussion_r80365720
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/json/JSONMapParserTest.java ---
@@ -56,7 +56,7 @@ public void testHappyPath() {
/**
{
- "collection" : { "blah" : 7, "blah2" : "foo" }
+ "collection" : { "blah" : 7, "blah2" : "foo", "bigblah" : { "innerBlah" : "baz", "reallyInnerBlah" : { "color" : "grey" }}}
--- End diff --
like
```
{
"collection" : {
"key" : "value"
},
"key" : "value"
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/271#discussion_r80365696
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/json/JSONMapParserTest.java ---
@@ -56,7 +56,7 @@ public void testHappyPath() {
/**
{
- "collection" : { "blah" : 7, "blah2" : "foo" }
+ "collection" : { "blah" : 7, "blah2" : "foo", "bigblah" : { "innerBlah" : "baz", "reallyInnerBlah" : { "color" : "grey" }}}
--- End diff --
I like it! Would you make one more test case that involves just a single map?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---