You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Girish N <gi...@gmail.com> on 2017/09/06 09:16:13 UTC
Exceptions in Parser, enrichments and indexing topology
Hi,
I am trying to parse the syslog I am getting below exceptions. Kindly help
to resolve the issue. Thanks
Sample Syslog-
Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
SYS_DELIMITED
<%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
2. Created a Parser config log.json
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "usr/metron/0.4.0/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}
3. Created a indexing config log.json
{
"hdfs" : {
"index": "log",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "log",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "log",
"batchSize": 5,
"enabled" : true
}
}
4. Pushed and Dumped using
usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
/usr/metron/0.4.0/config/zookeeper
usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
5. Then started the parser topology , getting the below exceptions in log
worker-artifacts (
usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
java.lang.IllegalStateException: Unhandled bulk errors in response:
{java.lang.IllegalArgumentException: Topic cannot be null=[source:
parserBolt:4, stream: error, id: {},
[{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
statement produced a null message. Original message was: Sep 6 12:41:02
exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
exza-ThinkPad-X240 systemd[1]: Started CUPS
Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {}
6. For the enrichment topology, getting the below exception.
2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
request.required.acks = 1 was supplied but isn't a known config.
2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
0.10.0.1
2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
a7a17cdec9eaa6c5
2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
enrichmentErrorOutputBolt:(1)
2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
System classloader
2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
threatIntelSplitBolt:(14)
2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
enrichmentSplitBolt:(3)
2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
hostEnrichmentBolt:(5)
2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
enrichmentJoinBolt:(2)
2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
org.apache.metron.jackson.core.JsonParseException: Unrecognized token
'enrichments': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
at
org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3466)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:854)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2867)
~[stormjar.jar:?]
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
~[stormjar.jar:?]
7. For the Indexing topology , getting below exception
com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
at
com.fasterxml.jackson.core.metron.elasticsearch.JsonParser._constructError(JsonParser.java:1581)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:737)
~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)
~[stormjar.jar:?]
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
~[stormjar.jar:?]
at
org.apache.metron.common.configuration.IndexingConfigurations.updateSensorIndexingConfig(IndexingConfigurations.java:52)
~[stormjar.jar:?]
Kindly help to resolve the issue.
Regards
Girish N
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Otto Fowler <ot...@gmail.com>.
https://issues.apache.org/jira/browse/METRON-1157
Feel free to review and add your comments.
It is important that ideas from the community get captured in jira, that is
the strength of the community effort and the apache/open source system.
It is what will help us to innovate.
We should be so lucky as to have too many jira issues with good ideas to
consider.
On September 6, 2017 at 07:58:31, ed d (ragdelaed@hotmail.com) wrote:
http://grokconstructor.appspot.com/do/match#result
Re: Exceptions in Parser, enrichments and indexing topology
Posted by ed d <ra...@hotmail.com>.
Well, that looks like a solid repository to me. No jira, will bookmark it and move on. :-)
Thanks Otto.
Sent from my Verizon 4G LTE Droid
On Sep 6, 2017 7:42 AM, Otto Fowler <ot...@gmail.com> wrote:
http://grokconstructor.appspot.com/do/match#result
mentions a lot of patterns.
Maybe open a jira?
On September 6, 2017 at 07:08:13, ed d (ragdelaed@hotmail.com<ma...@hotmail.com>) wrote:
Does anyone know of a solid repository of Grok patterns that match standard traffic patterns, like Cisco (non-ASA), JunOS, McAfee (all flavors), syslog, CheckPoint, etc?
Maybe a repository sectioned into tech, like Firewalls, Endpoints, Network, VPN, and other security tools? Is there a place in Metron we can start dumping some Grok filters?
I know there are many parsers already built, but it would be nice to have another method in case the parser does not fit exactly, or we need a non parser solution.
________________________________
From: Girish N <gi...@gmail.com>>
Sent: Wednesday, September 6, 2017 7:02 AM
To: dev@metron.apache.org<ma...@metron.apache.org>
Cc: Otto Fowler
Subject: Re: Exceptions in Parser, enrichments and indexing topology
Hello,
Grok pattern was wrong, changed it as below and tested and got a match.
Sample Log - Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
Grok pattern 2 - %{SYSLOGLINE}
Tried parsing with both the grok patterns, but still facing the same
exceptions.
Regards
Girish N
On Wed, Sep 6, 2017 at 4:17 PM, Anand Subramanian <
asubramanian@hortonworks.com<ma...@hortonworks.com>> wrote:
> Hey Girish,
>
> Can you validate using http://grokconstructor.appspot.com/do/match that
Test grok patterns - Grok Constructor<http://grokconstructor.appspot.com/do/match>
grokconstructor.appspot.com<http://grokconstructor.appspot.com>
This tries to parse a set of given logfile lines with a given grok regular expression (based on Oniguruma regular expressions) and prints the matches for ...
> you can parse the sample message using the pattern? I tried with your
> example and it did not work.
>
> -Anand
>
>
>
> On 9/6/17, 4:06 PM, "Girish N" <gi...@gmail.com>> wrote:
>
> >Thanks for your response Otto Fowler,
> >
> >I tried with the below config. Still the same exception. Kindly let me
> know
> >if anything else has to be changed.
> >
> >{
> >"parserClassName": "org.apache.metron.parsers.GrokParser",
> >"sensorTopic": "log",
> >"parserConfig": {
> >"grokPath": "/patterns/log",
> >"patternLabel": "SYS_DELIMITED"
> >}}
> >
> >java.lang.IllegalStateException: Unhandled bulk errors in response:
> >{java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >parserBolt:4, stream: error, id: {},
> >[{"exception":"java.lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {} . Check the pattern at: \/patterns\/log on
> >Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> >Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {}
> >
> >
> >Regards
> >Girish N
> >
> >
> >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com>>
> wrote:
> >
> >> Change the grokPath to match below and try.
> >>
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >>
> >> This path is not an absolute path.
> >>
> >> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
> >> com) wrote:
> >>
> >> Hi,
> >>
> >> I am trying to parse the syslog I am getting below exceptions. Kindly
> help
> >> to resolve the issue. Thanks
> >>
> >>
> >> Sample Syslog-
> >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
> >>
> >>
> >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> >> SYS_DELIMITED
> >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY}
> )?
> >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
> >>
> >> 2. Created a Parser config log.json
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "usr/metron/0.4.0/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >> 3. Created a indexing config log.json
> >> {
> >> "hdfs" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "elasticsearch" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "solr" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> }
> >> }
> >>
> >> 4. Pushed and Dumped using
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> >> /usr/metron/0.4.0/config/zookeeper
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
> >>
> >> 5. Then started the parser topology , getting the below exceptions in
> log
> >> worker-artifacts (
> >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
> >>
> >> java.lang.IllegalStateException: Unhandled bulk errors in response:
> >> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >> parserBolt:4, stream: error, id: {},
> >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> >> statement produced a null message. Original message was: Sep 6 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6
> 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS
> >> Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >>
> >> Grok parser Error: Grok statement produced a null message. Original
> >> message
> >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >> and the parsed message was: {}
> >>
> >> 6. For the enrichment topology, getting the below exception.
> >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The
> configuration
> >> request.required.acks = 1 was supplied but isn't a known config.
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> >> 0.10.0.1
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> >> a7a17cdec9eaa6c5
> >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentErrorOutputBolt:(1)
> >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> >> System classloader
> >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> >> threatIntelSplitBolt:(14)
> >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentSplitBolt:(3)
> >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> >> hostEnrichmentBolt:(5)
> >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentJoinBolt:(2)
> >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> >> 'enrichments': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> >> at
> >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.
> java:1586)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.base.ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:521)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(
> UTF8StreamJsonParser.java:748)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> initForReading(ObjectMapper.java:3847)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> readMapAndClose(ObjectMapper.java:3792)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper.
> >> readValue(ObjectMapper.java:2867)
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> >> ~[stormjar.jar:?]
> >>
> >> 7. For the Indexing topology , getting below exception
> >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> >> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.
> >> JsonParser._constructError(JsonParser.java:1581)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.base.
> ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:533)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:
> 3451)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:
> 2610)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._nextTokenNotInObject(
> UTF8StreamJsonParser.java:841)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser.
> >> nextToken(UTF8StreamJsonParser.java:737)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2874)
> >>
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.common.configuration.IndexingConfigurations.
> >> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> >> ~[stormjar.jar:?]
> >>
> >>
> >> Kindly help to resolve the issue.
> >>
> >> Regards
> >> Girish N
> >>
> >>
>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Otto Fowler <ot...@gmail.com>.
http://grokconstructor.appspot.com/do/match#result
mentions a lot of patterns.
Maybe open a jira?
On September 6, 2017 at 07:08:13, ed d (ragdelaed@hotmail.com) wrote:
Does anyone know of a solid repository of Grok patterns that match standard
traffic patterns, like Cisco (non-ASA), JunOS, McAfee (all flavors),
syslog, CheckPoint, etc?
Maybe a repository sectioned into tech, like Firewalls, Endpoints, Network,
VPN, and other security tools? Is there a place in Metron we can start
dumping some Grok filters?
I know there are many parsers already built, but it would be nice to have
another method in case the parser does not fit exactly, or we need a non
parser solution.
------------------------------
*From:* Girish N <gi...@gmail.com>
*Sent:* Wednesday, September 6, 2017 7:02 AM
*To:* dev@metron.apache.org
*Cc:* Otto Fowler
*Subject:* Re: Exceptions in Parser, enrichments and indexing topology
Hello,
Grok pattern was wrong, changed it as below and tested and got a match.
Sample Log - Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
Grok pattern 2 - %{SYSLOGLINE}
Tried parsing with both the grok patterns, but still facing the same
exceptions.
Regards
Girish N
On Wed, Sep 6, 2017 at 4:17 PM, Anand Subramanian <
asubramanian@hortonworks.com> wrote:
> Hey Girish,
>
> Can you validate using http://grokconstructor.appspot.com/do/match that
Test grok patterns - Grok Constructor
<http://grokconstructor.appspot.com/do/match>
grokconstructor.appspot.com
This tries to parse a set of given logfile lines with a given grok regular
expression (based on Oniguruma regular expressions) and prints the matches
for ...
> you can parse the sample message using the pattern? I tried with your
> example and it did not work.
>
> -Anand
>
>
>
> On 9/6/17, 4:06 PM, "Girish N" <gi...@gmail.com> wrote:
>
> >Thanks for your response Otto Fowler,
> >
> >I tried with the below config. Still the same exception. Kindly let me
> know
> >if anything else has to be changed.
> >
> >{
> >"parserClassName": "org.apache.metron.parsers.GrokParser",
> >"sensorTopic": "log",
> >"parserConfig": {
> >"grokPath": "/patterns/log",
> >"patternLabel": "SYS_DELIMITED"
> >}}
> >
> >java.lang.IllegalStateException: Unhandled bulk errors in response:
> >{java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >parserBolt:4, stream: error, id: {},
> >[{"exception":"java.lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {} . Check the pattern at: \/patterns\/log on
> >Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> >Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {}
> >
> >
> >Regards
> >Girish N
> >
> >
> >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com>
> wrote:
> >
> >> Change the grokPath to match below and try.
> >>
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >>
> >> This path is not an absolute path.
> >>
> >> On September 6, 2017 at 05:16:19, Girish N
(giri.narasimha.murthy@gmail.
> >> com) wrote:
> >>
> >> Hi,
> >>
> >> I am trying to parse the syslog I am getting below exceptions. Kindly
> help
> >> to resolve the issue. Thanks
> >>
> >>
> >> Sample Syslog-
> >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
> >>
> >>
> >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> >> SYS_DELIMITED
> >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY}
> )?
> >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
> >>
> >> 2. Created a Parser config log.json
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "usr/metron/0.4.0/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >> 3. Created a indexing config log.json
> >> {
> >> "hdfs" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "elasticsearch" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "solr" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> }
> >> }
> >>
> >> 4. Pushed and Dumped using
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> >> /usr/metron/0.4.0/config/zookeeper
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
> >>
> >> 5. Then started the parser topology , getting the below exceptions in
> log
> >> worker-artifacts (
> >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
> >>
> >> java.lang.IllegalStateException: Unhandled bulk errors in response:
> >> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >> parserBolt:4, stream: error, id: {},
> >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> >> statement produced a null message. Original message was: Sep 6 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6
> 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS
> >> Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >>
> >> Grok parser Error: Grok statement produced a null message. Original
> >> message
> >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >> and the parsed message was: {}
> >>
> >> 6. For the enrichment topology, getting the below exception.
> >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The
> configuration
> >> request.required.acks = 1 was supplied but isn't a known config.
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> >> 0.10.0.1
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> >> a7a17cdec9eaa6c5
> >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentErrorOutputBolt:(1)
> >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> >> System classloader
> >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> >> threatIntelSplitBolt:(14)
> >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentSplitBolt:(3)
> >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> >> hostEnrichmentBolt:(5)
> >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentJoinBolt:(2)
> >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> >> 'enrichments': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> >> at
> >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.
> java:1586)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.base.ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:521)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(
> UTF8StreamJsonParser.java:748)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> initForReading(ObjectMapper.java:3847)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> readMapAndClose(ObjectMapper.java:3792)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper.
> >> readValue(ObjectMapper.java:2867)
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> >> ~[stormjar.jar:?]
> >>
> >> 7. For the Indexing topology , getting below exception
> >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> >> Unrecognized token 'indexing': was expecting ('true', 'false' or
'null')
> >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.
> >> JsonParser._constructError(JsonParser.java:1581)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.base.
> ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:533)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:
> 3451)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:
> 2610)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._nextTokenNotInObject(
> UTF8StreamJsonParser.java:841)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser.
> >> nextToken(UTF8StreamJsonParser.java:737)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2874)
> >>
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.common.configuration.IndexingConfigurations.
> >> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> >> ~[stormjar.jar:?]
> >>
> >>
> >> Kindly help to resolve the issue.
> >>
> >> Regards
> >> Girish N
> >>
> >>
>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by ed d <ra...@hotmail.com>.
Does anyone know of a solid repository of Grok patterns that match standard traffic patterns, like Cisco (non-ASA), JunOS, McAfee (all flavors), syslog, CheckPoint, etc?
Maybe a repository sectioned into tech, like Firewalls, Endpoints, Network, VPN, and other security tools? Is there a place in Metron we can start dumping some Grok filters?
I know there are many parsers already built, but it would be nice to have another method in case the parser does not fit exactly, or we need a non parser solution.
________________________________
From: Girish N <gi...@gmail.com>
Sent: Wednesday, September 6, 2017 7:02 AM
To: dev@metron.apache.org
Cc: Otto Fowler
Subject: Re: Exceptions in Parser, enrichments and indexing topology
Hello,
Grok pattern was wrong, changed it as below and tested and got a match.
Sample Log - Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
Grok pattern 2 - %{SYSLOGLINE}
Tried parsing with both the grok patterns, but still facing the same
exceptions.
Regards
Girish N
On Wed, Sep 6, 2017 at 4:17 PM, Anand Subramanian <
asubramanian@hortonworks.com> wrote:
> Hey Girish,
>
> Can you validate using http://grokconstructor.appspot.com/do/match that
Test grok patterns - Grok Constructor<http://grokconstructor.appspot.com/do/match>
grokconstructor.appspot.com
This tries to parse a set of given logfile lines with a given grok regular expression (based on Oniguruma regular expressions) and prints the matches for ...
> you can parse the sample message using the pattern? I tried with your
> example and it did not work.
>
> -Anand
>
>
>
> On 9/6/17, 4:06 PM, "Girish N" <gi...@gmail.com> wrote:
>
> >Thanks for your response Otto Fowler,
> >
> >I tried with the below config. Still the same exception. Kindly let me
> know
> >if anything else has to be changed.
> >
> >{
> >"parserClassName": "org.apache.metron.parsers.GrokParser",
> >"sensorTopic": "log",
> >"parserConfig": {
> >"grokPath": "/patterns/log",
> >"patternLabel": "SYS_DELIMITED"
> >}}
> >
> >java.lang.IllegalStateException: Unhandled bulk errors in response:
> >{java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >parserBolt:4, stream: error, id: {},
> >[{"exception":"java.lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {} . Check the pattern at: \/patterns\/log on
> >Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> >Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {}
> >
> >
> >Regards
> >Girish N
> >
> >
> >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com>
> wrote:
> >
> >> Change the grokPath to match below and try.
> >>
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >>
> >> This path is not an absolute path.
> >>
> >> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
> >> com) wrote:
> >>
> >> Hi,
> >>
> >> I am trying to parse the syslog I am getting below exceptions. Kindly
> help
> >> to resolve the issue. Thanks
> >>
> >>
> >> Sample Syslog-
> >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
> >>
> >>
> >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> >> SYS_DELIMITED
> >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY}
> )?
> >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
> >>
> >> 2. Created a Parser config log.json
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "usr/metron/0.4.0/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >> 3. Created a indexing config log.json
> >> {
> >> "hdfs" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "elasticsearch" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "solr" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> }
> >> }
> >>
> >> 4. Pushed and Dumped using
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> >> /usr/metron/0.4.0/config/zookeeper
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
> >>
> >> 5. Then started the parser topology , getting the below exceptions in
> log
> >> worker-artifacts (
> >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
> >>
> >> java.lang.IllegalStateException: Unhandled bulk errors in response:
> >> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >> parserBolt:4, stream: error, id: {},
> >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> >> statement produced a null message. Original message was: Sep 6 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6
> 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS
> >> Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >>
> >> Grok parser Error: Grok statement produced a null message. Original
> >> message
> >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >> and the parsed message was: {}
> >>
> >> 6. For the enrichment topology, getting the below exception.
> >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The
> configuration
> >> request.required.acks = 1 was supplied but isn't a known config.
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> >> 0.10.0.1
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> >> a7a17cdec9eaa6c5
> >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentErrorOutputBolt:(1)
> >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> >> System classloader
> >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> >> threatIntelSplitBolt:(14)
> >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentSplitBolt:(3)
> >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> >> hostEnrichmentBolt:(5)
> >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentJoinBolt:(2)
> >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> >> 'enrichments': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> >> at
> >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.
> java:1586)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.base.ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:521)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(
> UTF8StreamJsonParser.java:748)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> initForReading(ObjectMapper.java:3847)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> readMapAndClose(ObjectMapper.java:3792)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper.
> >> readValue(ObjectMapper.java:2867)
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> >> ~[stormjar.jar:?]
> >>
> >> 7. For the Indexing topology , getting below exception
> >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> >> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.
> >> JsonParser._constructError(JsonParser.java:1581)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.base.
> ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:533)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:
> 3451)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:
> 2610)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._nextTokenNotInObject(
> UTF8StreamJsonParser.java:841)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser.
> >> nextToken(UTF8StreamJsonParser.java:737)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2874)
> >>
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.common.configuration.IndexingConfigurations.
> >> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> >> ~[stormjar.jar:?]
> >>
> >>
> >> Kindly help to resolve the issue.
> >>
> >> Regards
> >> Girish N
> >>
> >>
>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Girish N <gi...@gmail.com>.
Hello,
Grok pattern was wrong, changed it as below and tested and got a match.
Sample Log - Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
Grok pattern 1 - %{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
Grok pattern 2 - %{SYSLOGLINE}
Tried parsing with both the grok patterns, but still facing the same
exceptions.
Regards
Girish N
On Wed, Sep 6, 2017 at 4:17 PM, Anand Subramanian <
asubramanian@hortonworks.com> wrote:
> Hey Girish,
>
> Can you validate using http://grokconstructor.appspot.com/do/match that
> you can parse the sample message using the pattern? I tried with your
> example and it did not work.
>
> -Anand
>
>
>
> On 9/6/17, 4:06 PM, "Girish N" <gi...@gmail.com> wrote:
>
> >Thanks for your response Otto Fowler,
> >
> >I tried with the below config. Still the same exception. Kindly let me
> know
> >if anything else has to be changed.
> >
> >{
> >"parserClassName": "org.apache.metron.parsers.GrokParser",
> >"sensorTopic": "log",
> >"parserConfig": {
> >"grokPath": "/patterns/log",
> >"patternLabel": "SYS_DELIMITED"
> >}}
> >
> >java.lang.IllegalStateException: Unhandled bulk errors in response:
> >{java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >parserBolt:4, stream: error, id: {},
> >[{"exception":"java.lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {} . Check the pattern at: \/patterns\/log on
> >Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> >Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >Grok parser Error: Grok statement produced a null message. Original
> message
> >was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >and the parsed message was: {}
> >
> >
> >Regards
> >Girish N
> >
> >
> >On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com>
> wrote:
> >
> >> Change the grokPath to match below and try.
> >>
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >>
> >> This path is not an absolute path.
> >>
> >> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
> >> com) wrote:
> >>
> >> Hi,
> >>
> >> I am trying to parse the syslog I am getting below exceptions. Kindly
> help
> >> to resolve the issue. Thanks
> >>
> >>
> >> Sample Syslog-
> >> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
> >>
> >>
> >> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> >> SYS_DELIMITED
> >> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY}
> )?
> >> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
> >>
> >> 2. Created a Parser config log.json
> >> {
> >> "parserClassName": "org.apache.metron.parsers.GrokParser",
> >> "sensorTopic": "log",
> >> "parserConfig": {
> >> "grokPath": "usr/metron/0.4.0/patterns/log",
> >> "patternLabel": "SYS_DELIMITED"
> >> }}
> >>
> >> 3. Created a indexing config log.json
> >> {
> >> "hdfs" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "elasticsearch" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> },
> >> "solr" : {
> >> "index": "log",
> >> "batchSize": 5,
> >> "enabled" : true
> >> }
> >> }
> >>
> >> 4. Pushed and Dumped using
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> >> /usr/metron/0.4.0/config/zookeeper
> >> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
> >>
> >> 5. Then started the parser topology , getting the below exceptions in
> log
> >> worker-artifacts (
> >> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
> >>
> >> java.lang.IllegalStateException: Unhandled bulk errors in response:
> >> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> >> parserBolt:4, stream: error, id: {},
> >> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> >> statement produced a null message. Original message was: Sep 6 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> >> message was: {} . Check the pattern at: \/patterns\/log on Sep 6
> 12:41:02
> >> exza-ThinkPad-X240 systemd[1]: Started CUPS
> >> Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> >>
> >> Grok parser Error: Grok statement produced a null message. Original
> >> message
> >> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.
> >> and the parsed message was: {}
> >>
> >> 6. For the enrichment topology, getting the below exception.
> >> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The
> configuration
> >> request.required.acks = 1 was supplied but isn't a known config.
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> >> 0.10.0.1
> >> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> >> a7a17cdec9eaa6c5
> >> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentErrorOutputBolt:(1)
> >> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> >> System classloader
> >> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> >> threatIntelSplitBolt:(14)
> >> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentSplitBolt:(3)
> >> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> >> hostEnrichmentBolt:(5)
> >> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> >> enrichmentJoinBolt:(2)
> >> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> >> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> >> 'enrichments': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> >> at
> >> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.
> java:1586)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.base.ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:521)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> >> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(
> UTF8StreamJsonParser.java:748)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> initForReading(ObjectMapper.java:3847)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper._
> >> readMapAndClose(ObjectMapper.java:3792)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.jackson.databind.ObjectMapper.
> >> readValue(ObjectMapper.java:2867)
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> >> ~[stormjar.jar:?]
> >>
> >> 7. For the Indexing topology , getting below exception
> >> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> >> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
> >> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.
> >> JsonParser._constructError(JsonParser.java:1581)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.base.
> ParserMinimalBase._
> >> reportError(ParserMinimalBase.java:533)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:
> 3451)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:
> 2610)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> >> UTF8StreamJsonParser._nextTokenNotInObject(
> UTF8StreamJsonParser.java:841)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser.
> >> nextToken(UTF8StreamJsonParser.java:737)
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> >>
> >> ~[stormjar.jar:?]
> >> at
> >> com.fasterxml.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2874)
> >>
> >> ~[stormjar.jar:?]
> >> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> >> ~[stormjar.jar:?]
> >> at
> >> org.apache.metron.common.configuration.IndexingConfigurations.
> >> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> >> ~[stormjar.jar:?]
> >>
> >>
> >> Kindly help to resolve the issue.
> >>
> >> Regards
> >> Girish N
> >>
> >>
>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Anand Subramanian <as...@hortonworks.com>.
Hey Girish,
Can you validate using http://grokconstructor.appspot.com/do/match that you can parse the sample message using the pattern? I tried with your example and it did not work.
-Anand
On 9/6/17, 4:06 PM, "Girish N" <gi...@gmail.com> wrote:
>Thanks for your response Otto Fowler,
>
>I tried with the below config. Still the same exception. Kindly let me know
>if anything else has to be changed.
>
>{
>"parserClassName": "org.apache.metron.parsers.GrokParser",
>"sensorTopic": "log",
>"parserConfig": {
>"grokPath": "/patterns/log",
>"patternLabel": "SYS_DELIMITED"
>}}
>
>java.lang.IllegalStateException: Unhandled bulk errors in response:
>{java.lang.IllegalArgumentException: Topic cannot be null=[source:
>parserBolt:4, stream: error, id: {},
>[{"exception":"java.lang.IllegalStateException:
>Grok parser Error: Grok statement produced a null message. Original message
>was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
>and the parsed message was: {} . Check the pattern at: \/patterns\/log on
>Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
>Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
>Grok parser Error: Grok statement produced a null message. Original message
>was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
>and the parsed message was: {}
>
>
>Regards
>Girish N
>
>
>On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com> wrote:
>
>> Change the grokPath to match below and try.
>>
>> {
>> "parserClassName": "org.apache.metron.parsers.GrokParser",
>> "sensorTopic": "log",
>> "parserConfig": {
>> "grokPath": "/patterns/log",
>> "patternLabel": "SYS_DELIMITED"
>> }}
>>
>>
>> This path is not an absolute path.
>>
>> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
>> com) wrote:
>>
>> Hi,
>>
>> I am trying to parse the syslog I am getting below exceptions. Kindly help
>> to resolve the issue. Thanks
>>
>>
>> Sample Syslog-
>> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
>>
>>
>> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
>> SYS_DELIMITED
>> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
>> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
>>
>> 2. Created a Parser config log.json
>> {
>> "parserClassName": "org.apache.metron.parsers.GrokParser",
>> "sensorTopic": "log",
>> "parserConfig": {
>> "grokPath": "usr/metron/0.4.0/patterns/log",
>> "patternLabel": "SYS_DELIMITED"
>> }}
>>
>> 3. Created a indexing config log.json
>> {
>> "hdfs" : {
>> "index": "log",
>> "batchSize": 5,
>> "enabled" : true
>> },
>> "elasticsearch" : {
>> "index": "log",
>> "batchSize": 5,
>> "enabled" : true
>> },
>> "solr" : {
>> "index": "log",
>> "batchSize": 5,
>> "enabled" : true
>> }
>> }
>>
>> 4. Pushed and Dumped using
>> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
>> /usr/metron/0.4.0/config/zookeeper
>> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
>>
>> 5. Then started the parser topology , getting the below exceptions in log
>> worker-artifacts (
>> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
>>
>> java.lang.IllegalStateException: Unhandled bulk errors in response:
>> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
>> parserBolt:4, stream: error, id: {},
>> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
>> statement produced a null message. Original message was: Sep 6 12:41:02
>> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
>> message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
>> exza-ThinkPad-X240 systemd[1]: Started CUPS
>> Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
>>
>> Grok parser Error: Grok statement produced a null message. Original
>> message
>> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
>> and the parsed message was: {}
>>
>> 6. For the enrichment topology, getting the below exception.
>> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
>> request.required.acks = 1 was supplied but isn't a known config.
>> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
>> 0.10.0.1
>> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
>> a7a17cdec9eaa6c5
>> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
>> enrichmentErrorOutputBolt:(1)
>> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
>> System classloader
>> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
>> threatIntelSplitBolt:(14)
>> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
>> enrichmentSplitBolt:(3)
>> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
>> hostEnrichmentBolt:(5)
>> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
>> enrichmentJoinBolt:(2)
>> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
>> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
>> 'enrichments': was expecting ('true', 'false' or 'null')
>> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
>> at
>> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586)
>>
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.base.ParserMinimalBase._
>> reportError(ParserMinimalBase.java:521)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
>> reportInvalidToken(UTF8StreamJsonParser.java:3466)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
>> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
>> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
>>
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.databind.ObjectMapper._
>> initForReading(ObjectMapper.java:3847)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.databind.ObjectMapper._
>> readMapAndClose(ObjectMapper.java:3792)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.jackson.databind.ObjectMapper.
>> readValue(ObjectMapper.java:2867)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
>> ~[stormjar.jar:?]
>>
>> 7. For the Indexing topology , getting below exception
>> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
>> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
>> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.
>> JsonParser._constructError(JsonParser.java:1581)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._
>> reportError(ParserMinimalBase.java:533)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.
>> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.
>> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)
>>
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.
>> UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.
>> nextToken(UTF8StreamJsonParser.java:737)
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
>>
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
>>
>> ~[stormjar.jar:?]
>> at
>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)
>>
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
>> ~[stormjar.jar:?]
>> at
>> org.apache.metron.common.configuration.IndexingConfigurations.
>> updateSensorIndexingConfig(IndexingConfigurations.java:52)
>> ~[stormjar.jar:?]
>>
>>
>> Kindly help to resolve the issue.
>>
>> Regards
>> Girish N
>>
>>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Otto Fowler <ot...@gmail.com>.
Have you tested that grok against that message?
On September 6, 2017 at 06:36:24, Girish N (giri.narasimha.murthy@gmail.com)
wrote:
Thanks for your response Otto Fowler,
I tried with the below config. Still the same exception. Kindly let me know
if anything else has to be changed.
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}
java.lang.IllegalStateException: Unhandled bulk errors in response:
{java.lang.IllegalArgumentException: Topic cannot be null=[source:
parserBolt:4, stream: error, id: {},
[{"exception":"java.lang.IllegalStateException:
Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {} . Check the pattern at: \/patterns\/log on
Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {}
Regards
Girish N
On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com> wrote:
> Change the grokPath to match below and try.
>
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "log",
> "parserConfig": {
> "grokPath": "/patterns/log",
> "patternLabel": "SYS_DELIMITED"
> }}
>
>
> This path is not an absolute path.
>
> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
> com) wrote:
>
> Hi,
>
> I am trying to parse the syslog I am getting below exceptions. Kindly help
> to resolve the issue. Thanks
>
>
> Sample Syslog-
> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
>
>
> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> SYS_DELIMITED
> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
>
> 2. Created a Parser config log.json
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "log",
> "parserConfig": {
> "grokPath": "usr/metron/0.4.0/patterns/log",
> "patternLabel": "SYS_DELIMITED"
> }}
>
> 3. Created a indexing config log.json
> {
> "hdfs" : {
> "index": "log",
> "batchSize": 5,
> "enabled" : true
> },
> "elasticsearch" : {
> "index": "log",
> "batchSize": 5,
> "enabled" : true
> },
> "solr" : {
> "index": "log",
> "batchSize": 5,
> "enabled" : true
> }
> }
>
> 4. Pushed and Dumped using
> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> /usr/metron/0.4.0/config/zookeeper
> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
>
> 5. Then started the parser topology , getting the below exceptions in log
> worker-artifacts (
> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
>
> java.lang.IllegalStateException: Unhandled bulk errors in response:
> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> parserBolt:4, stream: error, id: {},
> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> statement produced a null message. Original message was: Sep 6 12:41:02
> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
> exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.","failed_sensor_type":"log","stack":"java.
> lang.IllegalStateException:
> Grok parser Error: Grok statement produced a null message. Original message
> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
> and the parsed message was: {}
>
> 6. For the enrichment topology, getting the below exception.
> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
> request.required.acks = 1 was supplied but isn't a known config.
> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> 0.10.0.1
> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> a7a17cdec9eaa6c5
> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> enrichmentErrorOutputBolt:(1)
> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> System classloader
> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> threatIntelSplitBolt:(14)
> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> enrichmentSplitBolt:(3)
> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> hostEnrichmentBolt:(5)
> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> enrichmentJoinBolt:(2)
> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> 'enrichments': was expecting ('true', 'false' or 'null')
> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> at
> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.
> java:1586)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.base.ParserMinimalBase._
> reportError(ParserMinimalBase.java:521)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(
> UTF8StreamJsonParser.java:748)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2867)
> ~[stormjar.jar:?]
> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> ~[stormjar.jar:?]
>
> 7. For the Indexing topology , getting below exception
> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.
> JsonParser._constructError(JsonParser.java:1581)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._
> reportError(ParserMinimalBase.java:533)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:
> 2610)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.
> nextToken(UTF8StreamJsonParser.java:737)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.
> java:3847)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.
> java:3792)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:
> 2874)
> ~[stormjar.jar:?]
> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.configuration.IndexingConfigurations.
> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> ~[stormjar.jar:?]
>
>
> Kindly help to resolve the issue.
>
> Regards
> Girish N
>
>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Girish N <gi...@gmail.com>.
Thanks for your response Otto Fowler,
I tried with the below config. Still the same exception. Kindly let me know
if anything else has to be changed.
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}
java.lang.IllegalStateException: Unhandled bulk errors in response:
{java.lang.IllegalArgumentException: Topic cannot be null=[source:
parserBolt:4, stream: error, id: {},
[{"exception":"java.lang.IllegalStateException:
Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {} . Check the pattern at: \/patterns\/log on
Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS
Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {}
Regards
Girish N
On Wed, Sep 6, 2017 at 3:45 PM, Otto Fowler <ot...@gmail.com> wrote:
> Change the grokPath to match below and try.
>
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "log",
> "parserConfig": {
> "grokPath": "/patterns/log",
> "patternLabel": "SYS_DELIMITED"
> }}
>
>
> This path is not an absolute path.
>
> On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.
> com) wrote:
>
> Hi,
>
> I am trying to parse the syslog I am getting below exceptions. Kindly help
> to resolve the issue. Thanks
>
>
> Sample Syslog-
> Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
>
>
> 1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
> SYS_DELIMITED
> <%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
> %{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
>
> 2. Created a Parser config log.json
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "log",
> "parserConfig": {
> "grokPath": "usr/metron/0.4.0/patterns/log",
> "patternLabel": "SYS_DELIMITED"
> }}
>
> 3. Created a indexing config log.json
> {
> "hdfs" : {
> "index": "log",
> "batchSize": 5,
> "enabled" : true
> },
> "elasticsearch" : {
> "index": "log",
> "batchSize": 5,
> "enabled" : true
> },
> "solr" : {
> "index": "log",
> "batchSize": 5,
> "enabled" : true
> }
> }
>
> 4. Pushed and Dumped using
> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> /usr/metron/0.4.0/config/zookeeper
> usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
>
> 5. Then started the parser topology , getting the below exceptions in log
> worker-artifacts (
> usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
>
> java.lang.IllegalStateException: Unhandled bulk errors in response:
> {java.lang.IllegalArgumentException: Topic cannot be null=[source:
> parserBolt:4, stream: error, id: {},
> [{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
> statement produced a null message. Original message was: Sep 6 12:41:02
> exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
> message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
> exza-ThinkPad-X240 systemd[1]: Started CUPS
> Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
>
> Grok parser Error: Grok statement produced a null message. Original
> message
> was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
> and the parsed message was: {}
>
> 6. For the enrichment topology, getting the below exception.
> 2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
> request.required.acks = 1 was supplied but isn't a known config.
> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
> 0.10.0.1
> 2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
> a7a17cdec9eaa6c5
> 2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
> enrichmentErrorOutputBolt:(1)
> 2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
> System classloader
> 2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
> threatIntelSplitBolt:(14)
> 2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
> enrichmentSplitBolt:(3)
> 2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
> hostEnrichmentBolt:(5)
> 2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
> enrichmentJoinBolt:(2)
> 2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
> org.apache.metron.jackson.core.JsonParseException: Unrecognized token
> 'enrichments': was expecting ('true', 'false' or 'null')
> at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
> at
> org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586)
>
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.base.ParserMinimalBase._
> reportError(ParserMinimalBase.java:521)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> reportInvalidToken(UTF8StreamJsonParser.java:3466)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser._
> nextTokenNotInObject(UTF8StreamJsonParser.java:854)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
>
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.ObjectMapper._
> initForReading(ObjectMapper.java:3847)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.ObjectMapper._
> readMapAndClose(ObjectMapper.java:3792)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.ObjectMapper.
> readValue(ObjectMapper.java:2867)
> ~[stormjar.jar:?]
> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
> ~[stormjar.jar:?]
>
> 7. For the Indexing topology , getting below exception
> com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
> Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
> at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.
> JsonParser._constructError(JsonParser.java:1581)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._
> reportError(ParserMinimalBase.java:533)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)
>
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.
> UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.
> nextToken(UTF8StreamJsonParser.java:737)
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
>
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
>
> ~[stormjar.jar:?]
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)
>
> ~[stormjar.jar:?]
> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.configuration.IndexingConfigurations.
> updateSensorIndexingConfig(IndexingConfigurations.java:52)
> ~[stormjar.jar:?]
>
>
> Kindly help to resolve the issue.
>
> Regards
> Girish N
>
>
Re: Exceptions in Parser, enrichments and indexing topology
Posted by Otto Fowler <ot...@gmail.com>.
Change the grokPath to match below and try.
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}
This path is not an absolute path.
On September 6, 2017 at 05:16:19, Girish N (giri.narasimha.murthy@gmail.com)
wrote:
Hi,
I am trying to parse the syslog I am getting below exceptions. Kindly help
to resolve the issue. Thanks
Sample Syslog-
Sep 6 14:13:42 exza-ThinkPad-X240 systemd[1]: Started Suspend.
1. Created a Grok pattern in path /usr/metron/0.4.0/patterns/log
SYS_DELIMITED
<%{NUMBER:queue_id}>+%{SYSLOGTIMESTAMP:timestamp2}(?:%{SYSLOGFACILITY} )?
%{IPORHOST} %{SYSLOGPROG}: %{GREEDYDATA:msg}
2. Created a Parser config log.json
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "log",
"parserConfig": {
"grokPath": "usr/metron/0.4.0/patterns/log",
"patternLabel": "SYS_DELIMITED"
}}
3. Created a indexing config log.json
{
"hdfs" : {
"index": "log",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "log",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "log",
"batchSize": 5,
"enabled" : true
}
}
4. Pushed and Dumped using
usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
/usr/metron/0.4.0/config/zookeeper
usr/metron/0.4.0/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
5. Then started the parser topology , getting the below exceptions in log
worker-artifacts (
usr/share/apache-storm/logs/workers-artifacts/log-1-1504683893/6700)
java.lang.IllegalStateException: Unhandled bulk errors in response:
{java.lang.IllegalArgumentException: Topic cannot be null=[source:
parserBolt:4, stream: error, id: {},
[{"exception":"java.lang.IllegalStateException: Grok parser Error: Grok
statement produced a null message. Original message was: Sep 6 12:41:02
exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler. and the parsed
message was: {} . Check the pattern at: \/patterns\/log on Sep 6 12:41:02
exza-ThinkPad-X240 systemd[1]: Started CUPS
Scheduler.","failed_sensor_type":"log","stack":"java.lang.IllegalStateException:
Grok parser Error: Grok statement produced a null message. Original message
was: Sep 6 12:41:02 exza-ThinkPad-X240 systemd[1]: Started CUPS Scheduler.
and the parsed message was: {}
6. For the enrichment topology, getting the below exception.
2017-09-06 07:46:16.249 o.a.k.c.p.ProducerConfig [WARN] The configuration
request.required.acks = 1 was supplied but isn't a known config.
2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka version :
0.10.0.1
2017-09-06 07:46:16.249 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId :
a7a17cdec9eaa6c5
2017-09-06 07:46:16.250 o.a.s.d.executor [INFO] Prepared bolt
enrichmentErrorOutputBolt:(1)
2017-09-06 07:46:16.704 o.a.m.c.d.f.r.BaseFunctionResolver [WARN] Using
System classloader
2017-09-06 07:46:16.811 o.a.s.d.executor [INFO] Prepared bolt
threatIntelSplitBolt:(14)
2017-09-06 07:46:16.813 o.a.s.d.executor [INFO] Prepared bolt
enrichmentSplitBolt:(3)
2017-09-06 07:46:16.815 o.a.s.d.executor [INFO] Prepared bolt
hostEnrichmentBolt:(5)
2017-09-06 07:46:16.818 o.a.s.d.executor [INFO] Prepared bolt
enrichmentJoinBolt:(2)
2017-09-06 07:46:16.832 o.a.c.f.r.c.TreeCache [ERROR]
org.apache.metron.jackson.core.JsonParseException: Unrecognized token
'enrichments': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@6624ac0e; line: 1, column: 23]
at
org.apache.metron.jackson.core.JsonParser._constructError(JsonParser.java:1586)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3466)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2625)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:854)
~[stormjar.jar:?]
at
org.apache.metron.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:748)
~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
~[stormjar.jar:?]
at
org.apache.metron.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2867)
~[stormjar.jar:?]
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:55)
~[stormjar.jar:?]
7. For the Indexing topology , getting below exception
com.fasterxml.jackson.core.metron.elasticsearch.JsonParseException:
Unrecognized token 'indexing': was expecting ('true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream@6c1f5065; line: 1, column: 17]
at
com.fasterxml.jackson.core.metron.elasticsearch.JsonParser._constructError(JsonParser.java:1581)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3451)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2610)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:841)
~[stormjar.jar:?]
at
com.fasterxml.jackson.core.metron.elasticsearch.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:737)
~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792)
~[stormjar.jar:?]
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)
~[stormjar.jar:?]
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:41)
~[stormjar.jar:?]
at
org.apache.metron.common.configuration.IndexingConfigurations.updateSensorIndexingConfig(IndexingConfigurations.java:52)
~[stormjar.jar:?]
Kindly help to resolve the issue.
Regards
Girish N