You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2018/05/18 14:02:36 UTC
Request for Comment on new Syslog 5424 Parsing library
There have been some issues and talk about they way we parse syslog, and
the deficiencies of our grok and regex based approaches, mainly not
supporting structured data as I recall.
I played around with it some and decided to try to write an Antlr grammar
based on the RFC 5424 spec BNF to parse valid syslogs.
I have chosen to create this in my own github org, and will be distributing
through bintray/mvn central down the line. I *may* end up doing PR’s to
Metron and Nifi around this but that is not definite.
If anyone is interested, I would really appreciate any review or feedback.
Also, if anyone has any ‘clean’ 5424 logs that they can safely contribute
to expand my test set, that would be much appreciated.
https://github.com/palindromicity/simple-syslog-5424
thanks
ottO
Re: Request for Comment on new Syslog 5424 Parsing library
Posted by Otto Fowler <ot...@gmail.com>.
I am open to adding new syslog parsers or parser ‘specifications’ as I have
termed them in. Possibly using grok in the background.
On May 21, 2018 at 07:03:40, Otto Fowler (ottobackwards@gmail.com) wrote:
Thanks Ahmed. At the moment, I’m only concerned with RFC 5424 formatted
syslog <https://tools.ietf.org/html/rfc5424>, especially the structured
data ( the data in the []).
Such as:
<14>1 2014–06–20T09:14:07+00:00 loggregator
d0602076-b14a–4c55–852a–981e7afeed38 DEA MSG–01 [exampleSDID@32473 iut=“3”
eventSource=“Application” eventID=“1011”][exampleSDID@32480 iut=4
eventSource=Other Application eventID=2022] Removing instance
On May 20, 2018 at 19:03:29, Ahmed Shah (ahmedshah@cmail.carleton.ca) wrote:
Hello,
If needed this is what our syslog config files look like and our GROK
statement (used with Metron 0.4.2)
Server side syslog config files (messages sent to syslog are passed on to
Kafka):
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForServer-Encypted/rsyslog.conf
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForServer-Encypted/00-GCRserverReciDionaea.conf
Client/honeypot side config file:
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForHP-Encrypted/00-GCRdionaeaHP.conf
GROK Statement:
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/README.md
-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>
________________________________
From: Casey Stella <ce...@gmail.com>
Sent: May 18, 2018 10:59 AM
To: dev@metron.apache.org
Subject: Re: Request for Comment on new Syslog 5424 Parsing library
Cool! I'd welcome a syslog parser!
On Fri, May 18, 2018 at 10:02 AM Otto Fowler <ot...@gmail.com>
wrote:
> There have been some issues and talk about they way we parse syslog, and
> the deficiencies of our grok and regex based approaches, mainly not
> supporting structured data as I recall.
> I played around with it some and decided to try to write an Antlr grammar
> based on the RFC 5424 spec BNF to parse valid syslogs.
>
> I have chosen to create this in my own github org, and will be
distributing
> through bintray/mvn central down the line. I *may* end up doing PR’s to
> Metron and Nifi around this but that is not definite.
>
> If anyone is interested, I would really appreciate any review or feedback.
> Also, if anyone has any ‘clean’ 5424 logs that they can safely contribute
> to expand my test set, that would be much appreciated.
>
> https://github.com/palindromicity/simple-syslog-5424
>
>
> thanks
> ottO
>
Re: Request for Comment on new Syslog 5424 Parsing library
Posted by Otto Fowler <ot...@gmail.com>.
Thanks Ahmed. At the moment, I’m only concerned with RFC 5424 formatted
syslog <https://tools.ietf.org/html/rfc5424>, especially the structured
data ( the data in the []).
Such as:
<14>1 2014–06–20T09:14:07+00:00 loggregator
d0602076-b14a–4c55–852a–981e7afeed38 DEA MSG–01 [exampleSDID@32473 iut=“3”
eventSource=“Application” eventID=“1011”][exampleSDID@32480 iut=4
eventSource=Other Application eventID=2022] Removing instance
On May 20, 2018 at 19:03:29, Ahmed Shah (ahmedshah@cmail.carleton.ca) wrote:
Hello,
If needed this is what our syslog config files look like and our GROK
statement (used with Metron 0.4.2)
Server side syslog config files (messages sent to syslog are passed on to
Kafka):
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForServer-Encypted/rsyslog.conf
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForServer-Encypted/00-GCRserverReciDionaea.conf
Client/honeypot side config file:
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForHP-Encrypted/00-GCRdionaeaHP.conf
GROK Statement:
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/README.md
-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>
________________________________
From: Casey Stella <ce...@gmail.com>
Sent: May 18, 2018 10:59 AM
To: dev@metron.apache.org
Subject: Re: Request for Comment on new Syslog 5424 Parsing library
Cool! I'd welcome a syslog parser!
On Fri, May 18, 2018 at 10:02 AM Otto Fowler <ot...@gmail.com>
wrote:
> There have been some issues and talk about they way we parse syslog, and
> the deficiencies of our grok and regex based approaches, mainly not
> supporting structured data as I recall.
> I played around with it some and decided to try to write an Antlr grammar
> based on the RFC 5424 spec BNF to parse valid syslogs.
>
> I have chosen to create this in my own github org, and will be
distributing
> through bintray/mvn central down the line. I *may* end up doing PR’s to
> Metron and Nifi around this but that is not definite.
>
> If anyone is interested, I would really appreciate any review or
feedback.
> Also, if anyone has any ‘clean’ 5424 logs that they can safely contribute
> to expand my test set, that would be much appreciated.
>
> https://github.com/palindromicity/simple-syslog-5424
>
>
> thanks
> ottO
>
Re: Request for Comment on new Syslog 5424 Parsing library
Posted by Ahmed Shah <Ah...@cmail.carleton.ca>.
Hello,
If needed this is what our syslog config files look like and our GROK statement (used with Metron 0.4.2)
Server side syslog config files (messages sent to syslog are passed on to Kafka):
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForServer-Encypted/rsyslog.conf
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForServer-Encypted/00-GCRserverReciDionaea.conf
Client/honeypot side config file:
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/configForHP-Encrypted/00-GCRdionaeaHP.conf
GROK Statement:
https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/SampleLogFiles/README.md
-Ahmed
_______________________________________________________________
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>
________________________________
From: Casey Stella <ce...@gmail.com>
Sent: May 18, 2018 10:59 AM
To: dev@metron.apache.org
Subject: Re: Request for Comment on new Syslog 5424 Parsing library
Cool! I'd welcome a syslog parser!
On Fri, May 18, 2018 at 10:02 AM Otto Fowler <ot...@gmail.com>
wrote:
> There have been some issues and talk about they way we parse syslog, and
> the deficiencies of our grok and regex based approaches, mainly not
> supporting structured data as I recall.
> I played around with it some and decided to try to write an Antlr grammar
> based on the RFC 5424 spec BNF to parse valid syslogs.
>
> I have chosen to create this in my own github org, and will be distributing
> through bintray/mvn central down the line. I *may* end up doing PR’s to
> Metron and Nifi around this but that is not definite.
>
> If anyone is interested, I would really appreciate any review or feedback.
> Also, if anyone has any ‘clean’ 5424 logs that they can safely contribute
> to expand my test set, that would be much appreciated.
>
> https://github.com/palindromicity/simple-syslog-5424
>
>
> thanks
> ottO
>
Re: Request for Comment on new Syslog 5424 Parsing library
Posted by Casey Stella <ce...@gmail.com>.
Cool! I'd welcome a syslog parser!
On Fri, May 18, 2018 at 10:02 AM Otto Fowler <ot...@gmail.com>
wrote:
> There have been some issues and talk about they way we parse syslog, and
> the deficiencies of our grok and regex based approaches, mainly not
> supporting structured data as I recall.
> I played around with it some and decided to try to write an Antlr grammar
> based on the RFC 5424 spec BNF to parse valid syslogs.
>
> I have chosen to create this in my own github org, and will be distributing
> through bintray/mvn central down the line. I *may* end up doing PR’s to
> Metron and Nifi around this but that is not definite.
>
> If anyone is interested, I would really appreciate any review or feedback.
> Also, if anyone has any ‘clean’ 5424 logs that they can safely contribute
> to expand my test set, that would be much appreciated.
>
> https://github.com/palindromicity/simple-syslog-5424
>
>
> thanks
> ottO
>