You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by nickwallen <gi...@git.apache.org> on 2018/11/15 21:31:04 UTC
[GitHub] metron pull request #1265: METRON-1874 Create a Parser Debugger
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1265
METRON-1874 Create a Parser Debugger
Users should be able to parse messages in a controlled environment like the Stellar REPL. This can help troubleshoot issues that are occurring at high velocity in a live Parser topology.
* This could help a user understand why a deployed parser is not working as they would expect.
* This can help a user to test their parser configuration before deploying to their live Metron cluster.
* This can help a user understand why a particular message is failing to parse successfully.
## Try It Out
### Parse a Message
1. Define the parser configuration.
```
[Stellar]>>> config := SHELL_EDIT()
{
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
"filterClassName":"org.apache.metron.parsers.filters.StellarFilter",
"sensorTopic":"bro"
}
```
1. Grab a message from the input topic to parse. You could also just mock-up a message that you would like to test.
```
[Stellar]>>> bro := KAFKA_GET('bro')
[{"http": {"ts":1542313125.807068,"uid":"CUrRne3iLIxXavQtci","id.orig_h"...
```
1. Initialize the parser. The parser keeps track of the number of successes and failures which can be useful when parsing a batch of messages.
```
[Stellar]>>> parser := PARSER_INIT("bro", config)
Parser{0 successful, 0 error(s)}
```
1. Parse the message.
```
[Stellar]>>> msgs := PARSER_PARSE(parser, bro)
[{"bro_timestamp":"1542313125.807068","method":"GET","ip_dst_port":8080,...
```
1. Review the successfully parsed message.
```
[Stellar]>>> LENGTH(msgs)
1
```
```
[Stellar]>>> msg := GET(msgs, 0)
[Stellar]>>> MAP_GET("guid", msg)
7f2e0c77-c58c-488e-b1ad-fbec10fb8182
```
```
[Stellar]>>> MAP_GET("timestamp", msg)
1542313125807
```
```
[Stellar]>>> MAP_GET("source.type", msg)
bro
```
1. The parser will tally the success.
```
[Stellar]>>> parser
Parser{1 successful, 0 error(s)}
```
### Parse Multiple Messages
1. Grab 5 raw input messages from Kafka.
```
[Stellar]>>> input := KAFKA_GET("bro", 5)
[{"dns": {"ts":1542313125.342913,"uid":"CmJWpN3Ynwsggof57e", ...
```
1. Parse the messages.
```
[Stellar]>>> msgs := PARSER_PARSE(parser, input)
[{"TTLs":[13888.0],"qclass_name":"C_INTERNET", ...
```
1. Review the parsed messages. There were 5 messages returned and each have a valid GUID as you would expect.
```
[Stellar]>>> LENGTH(msgs)
5
```
```
[Stellar]>>> MAP(msgs, m -> MAP_GET("guid", m))
[3b40ab62-ab6a-4dff-86c5-f35cdb2b01ea, 3b5826a7-f2d4-4df2-a28a-ab1f66037b4b, 9fc5f794-26f6-464f-bb99-05fb649ea465, c7162bee-01f9-4cc2-8e26-13101bc22ac1, b86dbb50-cb1d-4889-87ee-3919bcce6fdb]
````
### Parse an Invalid Message
1. Mock-up a message that will fail to parse.
```
[Stellar]>>> invalid := "{invalid>"
{invalid>
```
1. Parse the invalid message. This will show you the actual exception that occurred along with return the error message that is pushed onto the error topic.
```
[Stellar]>>> errors := PARSER_PARSE(parser, invalid)
2018-11-15 20:29:01 ERROR BasicBroParser:144 - Unable to parse Message: {invalid>
Unexpected character (i) at position 1.
at org.json.simple.parser.Yylex.yylex(Yylex.java:610)
at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
at org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)
at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)
at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)
at org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)
at org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)
at org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)
at org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)
at org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
at org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[{"exception":"java.lang.IllegalStateException: Unable to parse Message: {invalid>","failed_sensor_type":"bro","stack":"java.lang.IllegalStateException: Unable to parse Message: {invalid>\n\tat org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)\n\tat org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)\n\tat org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)\n\tat org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)\n\tat org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)\n\tat java.util.
stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\n\tat java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)\n\tat org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)\n\tat org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)\n\tat org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)\n\tat org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)\n\tat org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)\n\tat org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)\n\tat org.apache.metron.stellar.c
ommon.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)\n\tat org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)\n\tat org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)\n\tat org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)\n\tat org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)\n\tat org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: Unexpected character (i) at position 1.\n\tat org.json.simple.parser.Yylex.yylex(Yylex.java:610)\n\tat org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)\n\ta
t org.json.simple.parser.JSONParser.parse(JSONParser.java:118)\n\tat org.json.simple.parser.JSONParser.parse(JSONParser.java:81)\n\tat org.json.simple.parser.JSONParser.parse(JSONParser.java:75)\n\tat org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)\n\tat org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)\n\t... 28 more\n","hostname":"node1","raw_message":"{invalid>","error_hash":"e547a79488545c912977781a8d556341b3263943fad484f4d4b87e3b6052eac2","error_type":"parser_error","guid":"a2e06d23-6cef-4f26-b0ba-52263764d4ef","message":"Unable to parse Message: {invalid>","source.type":"error","timestamp":1542313741271}]
```
1. Review the details of the error.
```
[Stellar]>>> error := GET(errors, 0)
[Stellar]>>> MAP_GET("raw_message", error)
{invalid>
[Stellar]>>> MAP_GET("message", error)
Unable to parse Message: {invalid>
[Stellar]>>> MAP_GET("stack", error)
java.lang.IllegalStateException: Unable to parse Message: {invalid>
at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:145)
at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:54)
at org.apache.metron.parsers.interfaces.MessageParser.parseOptionalResult(MessageParser.java:67)
at org.apache.metron.parsers.ParserRunnerImpl.execute(ParserRunnerImpl.java:146)
at org.apache.metron.management.StellarParserRunner.lambda$doParse$3(StellarParserRunner.java:76)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.metron.management.StellarParserRunner.doParse(StellarParserRunner.java:77)
at org.apache.metron.management.StellarParserRunner.parse(StellarParserRunner.java:62)
at org.apache.metron.management.ParserFunctions$ParseFunction.apply(ParserFunctions.java:98)
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
at org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: Unexpected character (i) at position 1.
at org.json.simple.parser.Yylex.yylex(Yylex.java:610)
at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
at org.apache.metron.parsers.bro.JSONCleaner.clean(JSONCleaner.java:49)
at org.apache.metron.parsers.bro.BasicBroParser.parse(BasicBroParser.java:68)
... 28 more
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1874
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1265.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1265
----
commit 98155b884a8039017a244c18bd20d13989ab69f7
Author: Nick Allen <ni...@...>
Date: 2018-10-18T23:38:47Z
Initial pass
commit 9aa0c366fdff0ef40c7fdb389e4f5dc0316d637b
Author: Nick Allen <ni...@...>
Date: 2018-11-15T19:10:49Z
Able to parse one or messages in the REPL
commit 7854a2f75c47b72c51150902c03458076c00d87b
Author: Nick Allen <ni...@...>
Date: 2018-11-15T21:28:22Z
Tallying successes and errors. Added config function.
----
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1265
Can we load files from disk? It would be nice to not have had to setup kafka etc.
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1265
I was also wanting to do a follow-on that would load the parser configuration from Zk instead of requiring you to define that. In some cases, you just want to work with whatever the current 'live' configuration is.
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1265
Then maybe a function to evaluate stellar
EVALUATE_TRANSFORM(config,PARSER_PARSE(parser, input)
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1265
> @ottobackwards: Can we test parser chains?
I don't think so at the moment. Aggregation/chaining involves multiple parsers, each reading/writing data from Kafka. These functions work at the level of 1 parser and 1 `ParserRunner`. Thanks to @justinleet for my quick education on the topic.
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1265
maybe that can be a follow on, it would be much better to load from disk -> split lines than to open an editor and cut and past from your sample log.
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1265
Can we test parser chains?
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1265
> @ottobackwards: Can we load files from disk? It would be nice to not have had to setup kafka etc.
There is no requirement to setup Kafka. All you have to do is create a String that contains your message. If you want to get that message from Kafka, create it yourself using `SHELL_EDIT`, or copy-paste, then it works all the same.
In this PR you cannot load from disk.
---
[GitHub] metron pull request #1265: METRON-1874 Create a Parser Debugger
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1265
---
[GitHub] metron pull request #1265: METRON-1874 Create a Parser Debugger
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1265#discussion_r235005842
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java ---
@@ -87,13 +87,13 @@ public boolean isError() {
protected transient Consumer<ParserRunnerResults> onSuccess;
protected transient Consumer<MetronError> onError;
- private HashSet<String> sensorTypes;
+ private Set<String> sensorTypes;
--- End diff --
Shouldn't this be serializable?
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1265
Love it. I tested it in full dev and worked great. +1
---
[GitHub] metron pull request #1265: METRON-1874 Create a Parser Debugger
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1265#discussion_r235108449
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java ---
@@ -87,13 +87,13 @@ public boolean isError() {
protected transient Consumer<ParserRunnerResults> onSuccess;
protected transient Consumer<MetronError> onError;
- private HashSet<String> sensorTypes;
+ private Set<String> sensorTypes;
--- End diff --
I backed this out. Good catch.
---
[GitHub] metron pull request #1265: METRON-1874 Create a Parser Debugger
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1265#discussion_r235049017
--- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java ---
@@ -87,13 +87,13 @@ public boolean isError() {
protected transient Consumer<ParserRunnerResults> onSuccess;
protected transient Consumer<MetronError> onError;
- private HashSet<String> sensorTypes;
+ private Set<String> sensorTypes;
--- End diff --
I did not intend for this change to be included in this PR. Let me back this out.
---
[GitHub] metron issue #1265: METRON-1874 Create a Parser Debugger
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1265
> @ottobackwards maybe that can be a follow on, it would be much better to load from disk -> split lines than to open an editor and cut and past from your sample log.
Yep, agreed. It could just be a general purpose function that reads the contents of a file into a String. (Assuming we don't already have something like that.)
---