You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Frank Horsfall <Fr...@cunet.carleton.ca> on 2017/09/13 14:36:12 UTC
Grok Parser issues
Morning all,
Is anyone else seeing this error?
After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use.
Test data
Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
Grok statement
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
I tested the pattern at the Grok site
http://grokconstructor.appspot.com/do/match#result
[cid:image003.jpg@01D32C7C.1BA3C1C0]
Added the pattern to hdfs
[hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
Dump of zookeeper
PARSER Config: blah
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "blah",
"parserConfig": {
"grokPath": "/apps/metron/patterns/dionaea",
"patternLabel": "blah_DELIMITED",
"timestampField": "timestamp"
}
}
INDEXING Config: blah
{
"elasticsearch": {
"index": "blah",
"batchSize": 5,
"enabled": true
},
"hdfs": {
"index": "blah",
"batchSize": 5,
"enabled" : true
},
"solr": {
"index": "blah",
"batchSize": 5,
"enabled" : false
}
}
ENRICHMENT Config: blah
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs.
java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more
Any ideas?
Kindest regards,
Frank
RE: Grok Parser issues
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Thanks Ryan
There seems to be a Grok Validator in the Management UI which would be cool if I could get it running.
.
Where would I create the directory so that I don’t receive the error ?
patterns/temp/admin/blah (No such file or directory)
I believe it is tied to the Metron temp grok path under the advanced settings in Metron
[cid:image003.png@01D32CA6.7EF1DE90]
Cheers,
Frank
From: Ryan Merriman [mailto:merrimanr@gmail.com]
Sent: Wednesday, September 13, 2017 11:17 AM
To: user@metron.apache.org
Subject: Re: Grok Parser issues
Yes
On Wed, Sep 13, 2017 at 10:02 AM, Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
Do you mean this one on port 4200?
[cid:image001.png@01D32CA5.62A36DE0]
From: Ryan Merriman [mailto:merrimanr@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 13, 2017 10:53 AM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Grok Parser issues
You're better off using our management UI, it runs the same code that the parser topology does. I would start small with just a couple expressions (something like "blah_DELIMITED %{MONTH:month}") and ensure you're at least getting back the month. Then you can incrementally add more on until you find out where your problem is.
Ryan
On Wed, Sep 13, 2017 at 9:36 AM, Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
Morning all,
Is anyone else seeing this error?
After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use.
Test data
Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
Grok statement
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
I tested the pattern at the Grok site
http://grokconstructor.appspot.com/do/match#result
[cid:image002.jpg@01D32CA5.62A36DE0]
Added the pattern to hdfs
[hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
Dump of zookeeper
PARSER Config: blah
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "blah",
"parserConfig": {
"grokPath": "/apps/metron/patterns/dionaea",
"patternLabel": "blah_DELIMITED",
"timestampField": "timestamp"
}
}
INDEXING Config: blah
{
"elasticsearch": {
"index": "blah",
"batchSize": 5,
"enabled": true
},
"hdfs": {
"index": "blah",
"batchSize": 5,
"enabled" : true
},
"solr": {
"index": "blah",
"batchSize": 5,
"enabled" : false
}
}
ENRICHMENT Config: blah
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs.
java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more
Any ideas?
Kindest regards,
Frank
Re: Grok Parser issues
Posted by Ryan Merriman <me...@gmail.com>.
Yes
On Wed, Sep 13, 2017 at 10:02 AM, Frank Horsfall <
Frankhorsfall@cunet.carleton.ca> wrote:
> Do you mean this one on port 4200?
>
>
>
>
>
> *From:* Ryan Merriman [mailto:merrimanr@gmail.com]
> *Sent:* Wednesday, September 13, 2017 10:53 AM
> *To:* user@metron.apache.org
> *Subject:* Re: Grok Parser issues
>
>
>
> You're better off using our management UI, it runs the same code that the
> parser topology does. I would start small with just a couple expressions
> (something like "blah_DELIMITED %{MONTH:month}") and ensure you're at
> least getting back the month. Then you can incrementally add more on until
> you find out where your problem is.
>
>
>
> Ryan
>
>
>
> On Wed, Sep 13, 2017 at 9:36 AM, Frank Horsfall <
> Frankhorsfall@cunet.carleton.ca> wrote:
>
> Morning all,
>
>
>
> Is anyone else seeing this error?
>
>
>
>
>
> After successfully going through the telemetry tutorial with squid (
> https://cwiki.apache.org/confluence/display/METRON/
> Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating
> a new telemetry based on a data set I wish to use.
>
>
>
> Test data
>
>
>
> Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|
> httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|
> httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|
> httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
>
> Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|
> httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
>
>
>
> Grok statement
>
>
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> I tested the pattern at the Grok site
>
> http://grokconstructor.appspot.com/do/match#result
>
>
>
>
>
> Added the pattern to hdfs
>
>
>
> [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> *Dump of zookeeper*
>
>
>
> PARSER Config: blah
>
> {
>
> "parserClassName": "org.apache.metron.parsers.GrokParser",
>
> "sensorTopic": "blah",
>
> "parserConfig": {
>
> "grokPath": "/apps/metron/patterns/dionaea",
>
> "patternLabel": "blah_DELIMITED",
>
> "timestampField": "timestamp"
>
> }
>
> }
>
>
>
>
>
> INDEXING Config: blah
>
> {
>
> "elasticsearch": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled": true
>
> },
>
> "hdfs": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : true
>
> },
>
> "solr": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : false
>
> }
>
> }
>
>
>
> ENRICHMENT Config: blah
>
> {
>
> "enrichment" : {
>
> "fieldMap":
>
> {
>
> "geo": ["ip_dst_addr", "ip_src_addr"],
>
> "host": ["host"]
>
> }
>
> },
>
> "threatIntel": {
>
> "fieldMap":
>
> {
>
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
>
> },
>
> "fieldToTypeMap":
>
> {
>
> "ip_src_addr" : ["malicious_ip"],
>
> "ip_dst_addr" : ["malicious_ip"]
>
> }
>
> }
>
> }
>
>
>
>
>
> *Nifi is set up and passes correctly. But when I get to the parserBolt
> an error occurs.*
>
>
>
> java.lang.IllegalStateException: Grok parser Error: Grok statement
> produced a null message. Original message was: Jul 28 00:13:24 device1
> devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at
> org.apache.metron.parsers.interfaces.MessageParser.
> parseOptional(MessageParser.java:45) at org.apache.metron.parsers.
> bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.
> executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Grok statement produced a null
> message. Original message was: Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea at org.apache.metron.parsers.
> GrokParser.parse(GrokParser.java:152) ... 12 more
>
>
>
>
>
> Any ideas?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>
>
>
>
>
>
>
RE: Grok Parser issues
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Do you mean this one on port 4200?
[cid:image001.png@01D32C7F.CF627000]
From: Ryan Merriman [mailto:merrimanr@gmail.com]
Sent: Wednesday, September 13, 2017 10:53 AM
To: user@metron.apache.org
Subject: Re: Grok Parser issues
You're better off using our management UI, it runs the same code that the parser topology does. I would start small with just a couple expressions (something like "blah_DELIMITED %{MONTH:month}") and ensure you're at least getting back the month. Then you can incrementally add more on until you find out where your problem is.
Ryan
On Wed, Sep 13, 2017 at 9:36 AM, Frank Horsfall <Fr...@cunet.carleton.ca>> wrote:
Morning all,
Is anyone else seeing this error?
After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use.
Test data
Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
Grok statement
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
I tested the pattern at the Grok site
http://grokconstructor.appspot.com/do/match#result
[cid:image002.jpg@01D32C7F.CF627000]
Added the pattern to hdfs
[hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
Dump of zookeeper
PARSER Config: blah
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "blah",
"parserConfig": {
"grokPath": "/apps/metron/patterns/dionaea",
"patternLabel": "blah_DELIMITED",
"timestampField": "timestamp"
}
}
INDEXING Config: blah
{
"elasticsearch": {
"index": "blah",
"batchSize": 5,
"enabled": true
},
"hdfs": {
"index": "blah",
"batchSize": 5,
"enabled" : true
},
"solr": {
"index": "blah",
"batchSize": 5,
"enabled" : false
}
}
ENRICHMENT Config: blah
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs.
java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more
Any ideas?
Kindest regards,
Frank
Re: Grok Parser issues
Posted by Ryan Merriman <me...@gmail.com>.
You're better off using our management UI, it runs the same code that the
parser topology does. I would start small with just a couple expressions
(something like "blah_DELIMITED %{MONTH:month}") and ensure you're at least
getting back the month. Then you can incrementally add more on until you
find out where your problem is.
Ryan
On Wed, Sep 13, 2017 at 9:36 AM, Frank Horsfall <
Frankhorsfall@cunet.carleton.ca> wrote:
> Morning all,
>
>
>
> Is anyone else seeing this error?
>
>
>
>
>
> After successfully going through the telemetry tutorial with squid (
> https://cwiki.apache.org/confluence/display/METRON/
> Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating
> a new telemetry based on a data set I wish to use.
>
>
>
> Test data
>
>
>
> Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|
> httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|
> httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|
> httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
>
> Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|
> httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
>
>
>
> Grok statement
>
>
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> I tested the pattern at the Grok site
>
> http://grokconstructor.appspot.com/do/match#result
>
>
>
>
>
> Added the pattern to hdfs
>
>
>
> [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> *Dump of zookeeper*
>
>
>
> PARSER Config: blah
>
> {
>
> "parserClassName": "org.apache.metron.parsers.GrokParser",
>
> "sensorTopic": "blah",
>
> "parserConfig": {
>
> "grokPath": "/apps/metron/patterns/dionaea",
>
> "patternLabel": "blah_DELIMITED",
>
> "timestampField": "timestamp"
>
> }
>
> }
>
>
>
>
>
> INDEXING Config: blah
>
> {
>
> "elasticsearch": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled": true
>
> },
>
> "hdfs": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : true
>
> },
>
> "solr": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : false
>
> }
>
> }
>
>
>
> ENRICHMENT Config: blah
>
> {
>
> "enrichment" : {
>
> "fieldMap":
>
> {
>
> "geo": ["ip_dst_addr", "ip_src_addr"],
>
> "host": ["host"]
>
> }
>
> },
>
> "threatIntel": {
>
> "fieldMap":
>
> {
>
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
>
> },
>
> "fieldToTypeMap":
>
> {
>
> "ip_src_addr" : ["malicious_ip"],
>
> "ip_dst_addr" : ["malicious_ip"]
>
> }
>
> }
>
> }
>
>
>
>
>
> *Nifi is set up and passes correctly. But when I get to the parserBolt
> an error occurs.*
>
>
>
> java.lang.IllegalStateException: Grok parser Error: Grok statement
> produced a null message. Original message was: Jul 28 00:13:24 device1
> devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at
> org.apache.metron.parsers.interfaces.MessageParser.
> parseOptional(MessageParser.java:45) at org.apache.metron.parsers.
> bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.
> executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Grok statement produced a null
> message. Original message was: Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea at org.apache.metron.parsers.
> GrokParser.parse(GrokParser.java:152) ... 12 more
>
>
>
>
>
> Any ideas?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>
>
>
>
>
Re: Grok Parser issues
Posted by Lehuede Sebastien <le...@gmail.com>.
Hello Frank,
I had same problem some weeks ago. For me first thing is to check if you don’t have an extra space at end of your grok parser line ?
Another thing is to follow same tutorial but use Metron GUI insteed of command Line.
Lehuédé Sebastien
> Le 13 sept. 2017 à 16:36, Frank Horsfall <Fr...@cunet.carleton.ca> a écrit :
>
> Morning all,
>
> Is anyone else seeing this error?
>
>
> After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use.
>
> Test data
>
> Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
> Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
> Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
>
> Grok statement
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
> I tested the pattern at the Grok site
> http://grokconstructor.appspot.com/do/match#result
>
> <image003.jpg>
>
> Added the pattern to hdfs
>
> [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
> Dump of zookeeper
>
> PARSER Config: blah
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "blah",
> "parserConfig": {
> "grokPath": "/apps/metron/patterns/dionaea",
> "patternLabel": "blah_DELIMITED",
> "timestampField": "timestamp"
> }
> }
>
>
> INDEXING Config: blah
> {
> "elasticsearch": {
> "index": "blah",
> "batchSize": 5,
> "enabled": true
> },
> "hdfs": {
> "index": "blah",
> "batchSize": 5,
> "enabled" : true
> },
> "solr": {
> "index": "blah",
> "batchSize": 5,
> "enabled" : false
> }
> }
>
> ENRICHMENT Config: blah
> {
> "enrichment" : {
> "fieldMap":
> {
> "geo": ["ip_dst_addr", "ip_src_addr"],
> "host": ["host"]
> }
> },
> "threatIntel": {
> "fieldMap":
> {
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
> },
> "fieldToTypeMap":
> {
> "ip_src_addr" : ["malicious_ip"],
> "ip_dst_addr" : ["malicious_ip"]
> }
> }
> }
>
>
> Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs.
>
> java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more
>
>
> Any ideas?
>
> Kindest regards,
> Frank
>
>
>
>
>
RE: Grok Parser issues
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
Hi Girish,
Based on a few recommendations I restarted the process using the Metron management UI. I am now working through the exercise of scaling down my data input string and Grok format in hopes of isolating the point where the error occurs.
Stay tuned,
Frank
From: Girish N [mailto:giri.narasimha.murthy@gmail.com]
Sent: Thursday, September 14, 2017 11:13 AM
To: user@metron.apache.org
Subject: RE: Grok Parser issues
Hi Frank
Are you able to overcome Grok Parser issue. If you are, could you please let me know how you were able to do it..Thanks
Regards
Girish N
On 13 Sep 2017 22:35, "Frank Horsfall" <Fr...@cunet.carleton.ca>> wrote:
I’ll get back to you once I have it running.
Frank
From: Girish N [mailto:giri.narasimha.murthy@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 13, 2017 12:26 PM
To: user@metron.apache.org<ma...@metron.apache.org>
Subject: Re: Grok Parser issues
Yes I am also facing the same issues.
On 13 Sep 2017 20:06, "Frank Horsfall" <Fr...@cunet.carleton.ca>> wrote:
Morning all,
Is anyone else seeing this error?
After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use.
Test data
Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
Grok statement
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
I tested the pattern at the Grok site
http://grokconstructor.appspot.com/do/match#result
[cid:image001.jpg@01D32D76.A4E3F4F0]
Added the pattern to hdfs
[hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
Dump of zookeeper
PARSER Config: blah
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "blah",
"parserConfig": {
"grokPath": "/apps/metron/patterns/dionaea",
"patternLabel": "blah_DELIMITED",
"timestampField": "timestamp"
}
}
INDEXING Config: blah
{
"elasticsearch": {
"index": "blah",
"batchSize": 5,
"enabled": true
},
"hdfs": {
"index": "blah",
"batchSize": 5,
"enabled" : true
},
"solr": {
"index": "blah",
"batchSize": 5,
"enabled" : false
}
}
ENRICHMENT Config: blah
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs.
java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more
Any ideas?
Kindest regards,
Frank
RE: Grok Parser issues
Posted by Girish N <gi...@gmail.com>.
Hi Frank
Are you able to overcome Grok Parser issue. If you are, could you please
let me know how you were able to do it..Thanks
Regards
Girish N
On 13 Sep 2017 22:35, "Frank Horsfall" <Fr...@cunet.carleton.ca>
wrote:
> I’ll get back to you once I have it running.
>
>
>
> Frank
>
>
>
>
>
>
>
>
>
>
>
> *From:* Girish N [mailto:giri.narasimha.murthy@gmail.com]
> *Sent:* Wednesday, September 13, 2017 12:26 PM
> *To:* user@metron.apache.org
> *Subject:* Re: Grok Parser issues
>
>
>
> Yes I am also facing the same issues.
>
>
>
> On 13 Sep 2017 20:06, "Frank Horsfall" <Fr...@cunet.carleton.ca>
> wrote:
>
> Morning all,
>
>
>
> Is anyone else seeing this error?
>
>
>
>
>
> After successfully going through the telemetry tutorial with squid (
> https://cwiki.apache.org/confluence/display/METRON/
> Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating
> a new telemetry based on a data set I wish to use.
>
>
>
> Test data
>
>
>
> Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|
> httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|
> httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|
> httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
>
> Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|
> httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
>
>
>
> Grok statement
>
>
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> I tested the pattern at the Grok site
>
> http://grokconstructor.appspot.com/do/match#result
>
>
>
>
>
> Added the pattern to hdfs
>
>
>
> [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> *Dump of zookeeper*
>
>
>
> PARSER Config: blah
>
> {
>
> "parserClassName": "org.apache.metron.parsers.GrokParser",
>
> "sensorTopic": "blah",
>
> "parserConfig": {
>
> "grokPath": "/apps/metron/patterns/dionaea",
>
> "patternLabel": "blah_DELIMITED",
>
> "timestampField": "timestamp"
>
> }
>
> }
>
>
>
>
>
> INDEXING Config: blah
>
> {
>
> "elasticsearch": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled": true
>
> },
>
> "hdfs": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : true
>
> },
>
> "solr": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : false
>
> }
>
> }
>
>
>
> ENRICHMENT Config: blah
>
> {
>
> "enrichment" : {
>
> "fieldMap":
>
> {
>
> "geo": ["ip_dst_addr", "ip_src_addr"],
>
> "host": ["host"]
>
> }
>
> },
>
> "threatIntel": {
>
> "fieldMap":
>
> {
>
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
>
> },
>
> "fieldToTypeMap":
>
> {
>
> "ip_src_addr" : ["malicious_ip"],
>
> "ip_dst_addr" : ["malicious_ip"]
>
> }
>
> }
>
> }
>
>
>
>
>
> *Nifi is set up and passes correctly. But when I get to the parserBolt
> an error occurs.*
>
>
>
> java.lang.IllegalStateException: Grok parser Error: Grok statement
> produced a null message. Original message was: Jul 28 00:13:24 device1
> devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at
> org.apache.metron.parsers.interfaces.MessageParser.
> parseOptional(MessageParser.java:45) at org.apache.metron.parsers.
> bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.
> executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Grok statement produced a null
> message. Original message was: Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea at org.apache.metron.parsers.
> GrokParser.parse(GrokParser.java:152) ... 12 more
>
>
>
>
>
> Any ideas?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>
>
>
>
>
>
RE: Grok Parser issues
Posted by Frank Horsfall <Fr...@cunet.carleton.ca>.
I’ll get back to you once I have it running.
Frank
From: Girish N [mailto:giri.narasimha.murthy@gmail.com]
Sent: Wednesday, September 13, 2017 12:26 PM
To: user@metron.apache.org
Subject: Re: Grok Parser issues
Yes I am also facing the same issues.
On 13 Sep 2017 20:06, "Frank Horsfall" <Fr...@cunet.carleton.ca>> wrote:
Morning all,
Is anyone else seeing this error?
After successfully going through the telemetry tutorial with squid (https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating a new telemetry based on a data set I wish to use.
Test data
Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
Grok statement
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
I tested the pattern at the Grok site
http://grokconstructor.appspot.com/do/match#result
[cid:image001.jpg@01D32C90.F5176420]
Added the pattern to hdfs
[hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
blah_DELIMITED %{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname} %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|%{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\|%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
Dump of zookeeper
PARSER Config: blah
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "blah",
"parserConfig": {
"grokPath": "/apps/metron/patterns/dionaea",
"patternLabel": "blah_DELIMITED",
"timestampField": "timestamp"
}
}
INDEXING Config: blah
{
"elasticsearch": {
"index": "blah",
"batchSize": 5,
"enabled": true
},
"hdfs": {
"index": "blah",
"batchSize": 5,
"enabled" : true
},
"solr": {
"index": "blah",
"batchSize": 5,
"enabled" : false
}
}
ENRICHMENT Config: blah
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
Nifi is set up and passes correctly. But when I get to the parserBolt an error occurs.
java.lang.IllegalStateException: Grok parser Error: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at org.apache.metron.parsers.interfaces.MessageParser.parseOptional(MessageParser.java:45) at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Grok statement produced a null message. Original message was: Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80 and the parsed message was: {} . Check the pattern at: /apps/metron/patterns/dionaea at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:152) ... 12 more
Any ideas?
Kindest regards,
Frank
Re: Grok Parser issues
Posted by Girish N <gi...@gmail.com>.
Yes I am also facing the same issues.
On 13 Sep 2017 20:06, "Frank Horsfall" <Fr...@cunet.carleton.ca>
wrote:
> Morning all,
>
>
>
> Is anyone else seeing this error?
>
>
>
>
>
> After successfully going through the telemetry tutorial with squid (
> https://cwiki.apache.org/confluence/display/METRON/
> Adding+a+New+Telemetry+Data+Source ) I started the exercise of creating
> a new telemetry based on a data set I wish to use.
>
>
>
> Test data
>
>
>
> Jul 28 00:13:24 device1 devicelogger.py: connection|5287|accept|tcp|
> httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5289|accept|tcp|
> httpd|1501200814.55|10.2.1.83|1126|10.2.1.99|80
>
> Jul 28 00:13:44 device1 devicelogger.py: connection|5288|accept|tcp|
> httpd|1501200808.64|10.2.1.83|1116|10.2.1.99|80
>
> Jul 28 00:59:49 device1 devicelogger.py: connection|5296|accept|tcp|
> httpd|1501203587.76|10.2.1.83|1556|10.2.1.99|80
>
>
>
> Grok statement
>
>
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> I tested the pattern at the Grok site
>
> http://grokconstructor.appspot.com/do/match#result
>
>
>
>
>
> Added the pattern to hdfs
>
>
>
> [hdfs@metn1 ~]$ hadoop fs -cat /apps/metron/patterns/blah
>
> blah_DELIMITED %{MONTH:month} %{MONTHDAY:day}
> %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} %{WORD:hostname}
> %{WORD:script}.%{WORD:extension}: %{WORD:connection}\|%{NUMBER:
> number}\|%{WORD:type}\|%{WORD:transport}\|%{WORD:protocol}\|
> %{NUMBER:timestamp}\|%{IP:ip_src_addr}\|%{NUMBER:src_port}\
> |%{IP:ip_dst_addr}\|%{NUMBER:dst_port}
>
>
>
>
>
> *Dump of zookeeper*
>
>
>
> PARSER Config: blah
>
> {
>
> "parserClassName": "org.apache.metron.parsers.GrokParser",
>
> "sensorTopic": "blah",
>
> "parserConfig": {
>
> "grokPath": "/apps/metron/patterns/dionaea",
>
> "patternLabel": "blah_DELIMITED",
>
> "timestampField": "timestamp"
>
> }
>
> }
>
>
>
>
>
> INDEXING Config: blah
>
> {
>
> "elasticsearch": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled": true
>
> },
>
> "hdfs": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : true
>
> },
>
> "solr": {
>
> "index": "blah",
>
> "batchSize": 5,
>
> "enabled" : false
>
> }
>
> }
>
>
>
> ENRICHMENT Config: blah
>
> {
>
> "enrichment" : {
>
> "fieldMap":
>
> {
>
> "geo": ["ip_dst_addr", "ip_src_addr"],
>
> "host": ["host"]
>
> }
>
> },
>
> "threatIntel": {
>
> "fieldMap":
>
> {
>
> "hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
>
> },
>
> "fieldToTypeMap":
>
> {
>
> "ip_src_addr" : ["malicious_ip"],
>
> "ip_dst_addr" : ["malicious_ip"]
>
> }
>
> }
>
> }
>
>
>
>
>
> *Nifi is set up and passes correctly. But when I get to the parserBolt
> an error occurs.*
>
>
>
> java.lang.IllegalStateException: Grok parser Error: Grok statement
> produced a null message. Original message was: Jul 28 00:13:24 device1
> devicelogger.py: connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea on Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> at org.apache.metron.parsers.GrokParser.parse(GrokParser.java:174) at
> org.apache.metron.parsers.interfaces.MessageParser.
> parseOptional(MessageParser.java:45) at org.apache.metron.parsers.
> bolt.ParserBolt.execute(ParserBolt.java:133) at org.apache.storm.daemon.
> executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Grok statement produced a null
> message. Original message was: Jul 28 00:13:24 device1 devicelogger.py:
> connection|5287|accept|tcp|httpd|1501200799.33|10.2.1.83|1084|10.2.1.99|80
> and the parsed message was: {} . Check the pattern at:
> /apps/metron/patterns/dionaea at org.apache.metron.parsers.
> GrokParser.parse(GrokParser.java:152) ... 12 more
>
>
>
>
>
> Any ideas?
>
>
>
> Kindest regards,
>
> Frank
>
>
>
>
>
>
>
>
>
>
>