You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Ali Nazemian <al...@gmail.com> on 2017/04/07 03:30:43 UTC
Develop some parsers
Hi all,
We are going to develop some parsers and have some contribution to the
community as a start point. I was wondering what the reason is behind
choosing Grok statements for some of the implementations and Java regex for
other ones? Is there any policy for that? Probably it would be better to
have the Java regex implementation due to performance concerns. However, I
am sure there is a reason that some of them have been implemented with
using Grok statements.
Regards,
Ali
Re: Develop some parsers
Posted by Nick Allen <ni...@nickallen.org>.
Nope. All you need is `vagrant up`.
On Sat, Apr 8, 2017 at 10:37 AM, Mark de Rijk <
mark.derijk@samarkconsulting.co.uk> wrote:
> Hi,
>
> I am trying to follow the dev environment setup instructions but I am
> missing the Vagrant Init steps and it skips right ahead to a vagrant up.
>
> So after Vagrant hostmanager it skips
> to
> cd metron-deployment/vagrant/full-dev-platform
> vagrant up
>
> Doesn’t it miss a vagrant init step there?
> If I am saying something stupid let me know though.
>
> Regards,
> Mark de Rijk
>
> On 07/April/2017 14:37 , "Ali Nazemian" <al...@gmail.com> wrote:
>
> Mark,
>
> Have you seen the following pages?
>
> https://cwiki.apache.org/confluence/display/METRON/
> Development+Guidelines
> https://cwiki.apache.org/confluence/display/METRON/Metron+Development+
> Environment+Setup+Instructions
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
>
>
> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> wrote:
>
> > To clarify I have written a lot of parsers for ArcSight over the
> years and
> > I would like to start contributing by developing parsers for the
> Metron
> > project.
> > Is there any documentation that will help me get started so I can
> start
> > cranking them out?
> > This is the first open source project I am looking to contribute to
> so
> > forgive me If I am asking stupid questions.
> >
> >
> >
> > On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ottobackwards@gmail.com
> >
> > wrote:
> >
> > > I also believe that grok parsers can be added through
> configuration only,
> > > without having to
> > > compile a parser.
> > >
> > > You can add a parser configuration targeting the basic grok parser
> and
> > just
> > > provide the grok
> > > parser rules.
> > >
> > >
> > > Just as a heads up, I’m currently working on the parsers to allow
> for
> > > writing and maintaining parsers
> > > outside the metron code tree, including providing a maven
> archetype.
> > This
> > > will allow you to create parsers
> > > without having to maintain a fork etc.
> > >
> > > Keep an eye out for METRON-258 as a PR on the list.
> > >
> > >
> > >
> > > On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> wrote:
> > >
> > > My understanding of Grok vs Java is to provide a tradeoff for ease
> of
> > > implementation vs performance (plus Java can also handle parsing
> that
> > would
> > > be too complicated for Grok.
> > >
> > > Grok is less performant and handles less complex parsing, but it's
> easy
> > to
> > > get things going and potentially maintained without writing and
> compiling
> > > Java.
> > >
> > > The Java implementation will be better for performance and can
> handle
> > more
> > > complicated parsing Grok can't.
> > >
> > > I believe the preference has generally been for Grok parsers if
> > > appropriate, otherwise Java parsers.
> > >
> > > Justin
> > >
> > > On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
> alinazemian@gmail.com>
> > > wrote:
> > >
> > > > Hi Mark,
> > > >
> > > > Yeah, that would be great. Can you please specify which devices
> you
> > have
> > > > developed so far?
> > > >
> > > > Cheers,
> > > > Ali
> > > >
> > > > On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <
> me.derijk@gmail.com>
> > > wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > I am a heavy arcsight user and I have written quite a few
> parsers
> > over
> > > > > time.
> > > > > I am new to contributing to open source projects however.
> > > > > @Ali, would you like to cooperate on development of some
> parsers?
> > > > >
> > > > > Kind Regards,
> > > > > Mark de Rijk
> > > > >
> > > > >
> > > > > > On 7 Apr 2017, at 04:30, Ali Nazemian <alinazemian@gmail.com
> >
> > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are going to develop some parsers and have some
> contribution to
> > > the
> > > > > > community as a start point. I was wondering what the reason
> is
> > behind
> > > > > > choosing Grok statements for some of the implementations and
> Java
> > > regex
> > > > > for
> > > > > > other ones? Is there any policy for that? Probably it would
> be
> > better
> > > > to
> > > > > > have the Java regex implementation due to performance
> concerns.
> > > > However,
> > > > > I
> > > > > > am sure there is a reason that some of them have been
> implemented
> > > with
> > > > > > using Grok statements.
> > > > > >
> > > > > > Regards,
> > > > > > Ali
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > > >
> > >
> >
>
>
>
> --
> A.Nazemian
>
>
>
Re: Develop some parsers
Posted by Mark de Rijk <ma...@samarkconsulting.co.uk>.
Hi,
I am trying to follow the dev environment setup instructions but I am missing the Vagrant Init steps and it skips right ahead to a vagrant up.
So after Vagrant hostmanager it skips
to
cd metron-deployment/vagrant/full-dev-platform
vagrant up
Doesn’t it miss a vagrant init step there?
If I am saying something stupid let me know though.
Regards,
Mark de Rijk
On 07/April/2017 14:37 , "Ali Nazemian" <al...@gmail.com> wrote:
Mark,
Have you seen the following pages?
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions
https://cwiki.apache.org/confluence/display/METRON/Community+Resources
On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com> wrote:
> To clarify I have written a lot of parsers for ArcSight over the years and
> I would like to start contributing by developing parsers for the Metron
> project.
> Is there any documentation that will help me get started so I can start
> cranking them out?
> This is the first open source project I am looking to contribute to so
> forgive me If I am asking stupid questions.
>
>
>
> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > I also believe that grok parsers can be added through configuration only,
> > without having to
> > compile a parser.
> >
> > You can add a parser configuration targeting the basic grok parser and
> just
> > provide the grok
> > parser rules.
> >
> >
> > Just as a heads up, I’m currently working on the parsers to allow for
> > writing and maintaining parsers
> > outside the metron code tree, including providing a maven archetype.
> This
> > will allow you to create parsers
> > without having to maintain a fork etc.
> >
> > Keep an eye out for METRON-258 as a PR on the list.
> >
> >
> >
> > On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
> >
> > My understanding of Grok vs Java is to provide a tradeoff for ease of
> > implementation vs performance (plus Java can also handle parsing that
> would
> > be too complicated for Grok.
> >
> > Grok is less performant and handles less complex parsing, but it's easy
> to
> > get things going and potentially maintained without writing and compiling
> > Java.
> >
> > The Java implementation will be better for performance and can handle
> more
> > complicated parsing Grok can't.
> >
> > I believe the preference has generally been for Grok parsers if
> > appropriate, otherwise Java parsers.
> >
> > Justin
> >
> > On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> > wrote:
> >
> > > Hi Mark,
> > >
> > > Yeah, that would be great. Can you please specify which devices you
> have
> > > developed so far?
> > >
> > > Cheers,
> > > Ali
> > >
> > > On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> > wrote:
> > >
> > > > Dear all,
> > > >
> > > > I am a heavy arcsight user and I have written quite a few parsers
> over
> > > > time.
> > > > I am new to contributing to open source projects however.
> > > > @Ali, would you like to cooperate on development of some parsers?
> > > >
> > > > Kind Regards,
> > > > Mark de Rijk
> > > >
> > > >
> > > > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > We are going to develop some parsers and have some contribution to
> > the
> > > > > community as a start point. I was wondering what the reason is
> behind
> > > > > choosing Grok statements for some of the implementations and Java
> > regex
> > > > for
> > > > > other ones? Is there any policy for that? Probably it would be
> better
> > > to
> > > > > have the Java regex implementation due to performance concerns.
> > > However,
> > > > I
> > > > > am sure there is a reason that some of them have been implemented
> > with
> > > > > using Grok statements.
> > > > >
> > > > > Regards,
> > > > > Ali
> > > >
> > >
> > >
> > >
> > > --
> > > A.Nazemian
> > >
> >
>
--
A.Nazemian
Re: Develop some parsers
Posted by Otto Fowler <ot...@gmail.com>.
Hopefully as we make producing parsers outside of metron easier ( if not
possible ) we will be able to get more traction around this.
On April 7, 2017 at 10:54:55, Mark de Rijk (me.derijk@gmail.com) wrote:
I do have datasets for some parsers separately such as for Infoblox.
Can we centralize the parser development somewhere as a document so for
instance we can vote on which ones the project needs first?
I would be happy to work on parsers and coordinate the retrieval of data
sets, anonymization/obfuscation and processing/generating the parsers.
Once I am more comfortable in Java and such I can then help with more
stuff.
Thanks for the welcome and I hope to start generating added value as soon
as possible.
On 07/April/2017 15:44 , "Nick Allen" <ni...@nickallen.org> wrote:
I think the primary limitation for parser contributors is that you need a
fair amount of good data to work with. Building a parser from a small
amount of test data will probably miss some corner cases that one would see
"in the wild." I don't think there are any right or wrong answers here
though. If you have access to good Infoblox data, for example, then start
there.
I would also suggest that you try building a parser with Grok expressions
first. You can do quite a bit with grok expressions. Get a feel for what
you can do with those, what the limitations are, and then transition into
Java parsers only as you see a need.
Really glad that you are joining the community. I look forward to your
contributions.
On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me...@gmail.com> wrote:
> Ali,
>
> Coding wise I am still getting my footing on and doing a Java course
> online.
> So building in unit tests I am afraid is a bit further away from me.
> For the parsers:
>
> - Infoblox Syslog
> - Cisco IOS Syslog
> - Unix Syslog
> - Alcatel Syslog
> - Sendmail Syslog
> - Blackberry Enterprise Server File
>
> I have also developed file and DB based connectors but as said I need to
> figure it out how to actually develop parsers for the Metron platform
> specifically.
>
> Is there a list of most requested devices documented somewhere so I can
> focus my efforts?
>
> Cheers,
> Mark
>
>
> On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Mark,
> >
> > Can you please specify the Parsers you are familiar with? We are
> > prioritizing the parser implementation, so your effort can help us to
> > target them faster.
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > I will review them this afternoon. Thanks for that.
> > >
> > > Sent from my iPhone
> > >
> > > > On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com>
wrote:
> > > >
> > > > Mark,
> > > >
> > > > Have you seen the following pages?
> > > >
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > Development+Guidelines
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Metron+Development+
> > > Environment+Setup+Instructions
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Community+Resources
> > > >
> > > >
> > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> > > wrote:
> > > >>
> > > >> To clarify I have written a lot of parsers for ArcSight over the
> years
> > > and
> > > >> I would like to start contributing by developing parsers for the
> > Metron
> > > >> project.
> > > >> Is there any documentation that will help me get started so I can
> > start
> > > >> cranking them out?
> > > >> This is the first open source project I am looking to contribute
to
> so
> > > >> forgive me If I am asking stupid questions.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <
> ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I also believe that grok parsers can be added through
configuration
> > > only,
> > > >>> without having to
> > > >>> compile a parser.
> > > >>>
> > > >>> You can add a parser configuration targeting the basic grok
parser
> > and
> > > >> just
> > > >>> provide the grok
> > > >>> parser rules.
> > > >>>
> > > >>>
> > > >>> Just as a heads up, I’m currently working on the parsers to allow
> for
> > > >>> writing and maintaining parsers
> > > >>> outside the metron code tree, including providing a maven
> archetype.
> > > >> This
> > > >>> will allow you to create parsers
> > > >>> without having to maintain a fork etc.
> > > >>>
> > > >>> Keep an eye out for METRON-258 as a PR on the list.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> > > wrote:
> > > >>>
> > > >>> My understanding of Grok vs Java is to provide a tradeoff for
ease
> of
> > > >>> implementation vs performance (plus Java can also handle parsing
> that
> > > >> would
> > > >>> be too complicated for Grok.
> > > >>>
> > > >>> Grok is less performant and handles less complex parsing, but
it's
> > easy
> > > >> to
> > > >>> get things going and potentially maintained without writing and
> > > compiling
> > > >>> Java.
> > > >>>
> > > >>> The Java implementation will be better for performance and can
> handle
> > > >> more
> > > >>> complicated parsing Grok can't.
> > > >>>
> > > >>> I believe the preference has generally been for Grok parsers if
> > > >>> appropriate, otherwise Java parsers.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
> alinazemian@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Mark,
> > > >>>>
> > > >>>> Yeah, that would be great. Can you please specify which devices
> you
> > > >> have
> > > >>>> developed so far?
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Ali
> > > >>>>
> > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.derijk@gmail.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Dear all,
> > > >>>>>
> > > >>>>> I am a heavy arcsight user and I have written quite a few
parsers
> > > >> over
> > > >>>>> time.
> > > >>>>> I am new to contributing to open source projects however.
> > > >>>>> @Ali, would you like to cooperate on development of some
parsers?
> > > >>>>>
> > > >>>>> Kind Regards,
> > > >>>>> Mark de Rijk
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> We are going to develop some parsers and have some
contribution
> to
> > > >>> the
> > > >>>>>> community as a start point. I was wondering what the reason is
> > > >> behind
> > > >>>>>> choosing Grok statements for some of the implementations and
> Java
> > > >>> regex
> > > >>>>> for
> > > >>>>>> other ones? Is there any policy for that? Probably it would be
> > > >> better
> > > >>>> to
> > > >>>>>> have the Java regex implementation due to performance
concerns.
> > > >>>> However,
> > > >>>>> I
> > > >>>>>> am sure there is a reason that some of them have been
> implemented
> > > >>> with
> > > >>>>>> using Grok statements.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Ali
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> A.Nazemian
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
Re: Develop some parsers
Posted by Mark de Rijk <me...@gmail.com>.
I do have datasets for some parsers separately such as for Infoblox.
Can we centralize the parser development somewhere as a document so for instance we can vote on which ones the project needs first?
I would be happy to work on parsers and coordinate the retrieval of data sets, anonymization/obfuscation and processing/generating the parsers.
Once I am more comfortable in Java and such I can then help with more stuff.
Thanks for the welcome and I hope to start generating added value as soon as possible.
On 07/April/2017 15:44 , "Nick Allen" <ni...@nickallen.org> wrote:
I think the primary limitation for parser contributors is that you need a
fair amount of good data to work with. Building a parser from a small
amount of test data will probably miss some corner cases that one would see
"in the wild." I don't think there are any right or wrong answers here
though. If you have access to good Infoblox data, for example, then start
there.
I would also suggest that you try building a parser with Grok expressions
first. You can do quite a bit with grok expressions. Get a feel for what
you can do with those, what the limitations are, and then transition into
Java parsers only as you see a need.
Really glad that you are joining the community. I look forward to your
contributions.
On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me...@gmail.com> wrote:
> Ali,
>
> Coding wise I am still getting my footing on and doing a Java course
> online.
> So building in unit tests I am afraid is a bit further away from me.
> For the parsers:
>
> - Infoblox Syslog
> - Cisco IOS Syslog
> - Unix Syslog
> - Alcatel Syslog
> - Sendmail Syslog
> - Blackberry Enterprise Server File
>
> I have also developed file and DB based connectors but as said I need to
> figure it out how to actually develop parsers for the Metron platform
> specifically.
>
> Is there a list of most requested devices documented somewhere so I can
> focus my efforts?
>
> Cheers,
> Mark
>
>
> On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Mark,
> >
> > Can you please specify the Parsers you are familiar with? We are
> > prioritizing the parser implementation, so your effort can help us to
> > target them faster.
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > I will review them this afternoon. Thanks for that.
> > >
> > > Sent from my iPhone
> > >
> > > > On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com> wrote:
> > > >
> > > > Mark,
> > > >
> > > > Have you seen the following pages?
> > > >
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > Development+Guidelines
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Metron+Development+
> > > Environment+Setup+Instructions
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Community+Resources
> > > >
> > > >
> > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> > > wrote:
> > > >>
> > > >> To clarify I have written a lot of parsers for ArcSight over the
> years
> > > and
> > > >> I would like to start contributing by developing parsers for the
> > Metron
> > > >> project.
> > > >> Is there any documentation that will help me get started so I can
> > start
> > > >> cranking them out?
> > > >> This is the first open source project I am looking to contribute to
> so
> > > >> forgive me If I am asking stupid questions.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <
> ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I also believe that grok parsers can be added through configuration
> > > only,
> > > >>> without having to
> > > >>> compile a parser.
> > > >>>
> > > >>> You can add a parser configuration targeting the basic grok parser
> > and
> > > >> just
> > > >>> provide the grok
> > > >>> parser rules.
> > > >>>
> > > >>>
> > > >>> Just as a heads up, I’m currently working on the parsers to allow
> for
> > > >>> writing and maintaining parsers
> > > >>> outside the metron code tree, including providing a maven
> archetype.
> > > >> This
> > > >>> will allow you to create parsers
> > > >>> without having to maintain a fork etc.
> > > >>>
> > > >>> Keep an eye out for METRON-258 as a PR on the list.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> > > wrote:
> > > >>>
> > > >>> My understanding of Grok vs Java is to provide a tradeoff for ease
> of
> > > >>> implementation vs performance (plus Java can also handle parsing
> that
> > > >> would
> > > >>> be too complicated for Grok.
> > > >>>
> > > >>> Grok is less performant and handles less complex parsing, but it's
> > easy
> > > >> to
> > > >>> get things going and potentially maintained without writing and
> > > compiling
> > > >>> Java.
> > > >>>
> > > >>> The Java implementation will be better for performance and can
> handle
> > > >> more
> > > >>> complicated parsing Grok can't.
> > > >>>
> > > >>> I believe the preference has generally been for Grok parsers if
> > > >>> appropriate, otherwise Java parsers.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
> alinazemian@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Mark,
> > > >>>>
> > > >>>> Yeah, that would be great. Can you please specify which devices
> you
> > > >> have
> > > >>>> developed so far?
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Ali
> > > >>>>
> > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.derijk@gmail.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Dear all,
> > > >>>>>
> > > >>>>> I am a heavy arcsight user and I have written quite a few parsers
> > > >> over
> > > >>>>> time.
> > > >>>>> I am new to contributing to open source projects however.
> > > >>>>> @Ali, would you like to cooperate on development of some parsers?
> > > >>>>>
> > > >>>>> Kind Regards,
> > > >>>>> Mark de Rijk
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> We are going to develop some parsers and have some contribution
> to
> > > >>> the
> > > >>>>>> community as a start point. I was wondering what the reason is
> > > >> behind
> > > >>>>>> choosing Grok statements for some of the implementations and
> Java
> > > >>> regex
> > > >>>>> for
> > > >>>>>> other ones? Is there any policy for that? Probably it would be
> > > >> better
> > > >>>> to
> > > >>>>>> have the Java regex implementation due to performance concerns.
> > > >>>> However,
> > > >>>>> I
> > > >>>>>> am sure there is a reason that some of them have been
> implemented
> > > >>> with
> > > >>>>>> using Grok statements.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Ali
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> A.Nazemian
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
Re: Develop some parsers
Posted by Mark de Rijk <ma...@samarkconsulting.co.uk>.
I rejoined the mailing lists with my correct email address for this.
Excuses for the confusion.
On 07/April/2017 15:44 , "Nick Allen" <ni...@nickallen.org> wrote:
I think the primary limitation for parser contributors is that you need a
fair amount of good data to work with. Building a parser from a small
amount of test data will probably miss some corner cases that one would see
"in the wild." I don't think there are any right or wrong answers here
though. If you have access to good Infoblox data, for example, then start
there.
I would also suggest that you try building a parser with Grok expressions
first. You can do quite a bit with grok expressions. Get a feel for what
you can do with those, what the limitations are, and then transition into
Java parsers only as you see a need.
Really glad that you are joining the community. I look forward to your
contributions.
On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me...@gmail.com> wrote:
> Ali,
>
> Coding wise I am still getting my footing on and doing a Java course
> online.
> So building in unit tests I am afraid is a bit further away from me.
> For the parsers:
>
> - Infoblox Syslog
> - Cisco IOS Syslog
> - Unix Syslog
> - Alcatel Syslog
> - Sendmail Syslog
> - Blackberry Enterprise Server File
>
> I have also developed file and DB based connectors but as said I need to
> figure it out how to actually develop parsers for the Metron platform
> specifically.
>
> Is there a list of most requested devices documented somewhere so I can
> focus my efforts?
>
> Cheers,
> Mark
>
>
> On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Mark,
> >
> > Can you please specify the Parsers you are familiar with? We are
> > prioritizing the parser implementation, so your effort can help us to
> > target them faster.
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > I will review them this afternoon. Thanks for that.
> > >
> > > Sent from my iPhone
> > >
> > > > On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com> wrote:
> > > >
> > > > Mark,
> > > >
> > > > Have you seen the following pages?
> > > >
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > Development+Guidelines
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Metron+Development+
> > > Environment+Setup+Instructions
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Community+Resources
> > > >
> > > >
> > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> > > wrote:
> > > >>
> > > >> To clarify I have written a lot of parsers for ArcSight over the
> years
> > > and
> > > >> I would like to start contributing by developing parsers for the
> > Metron
> > > >> project.
> > > >> Is there any documentation that will help me get started so I can
> > start
> > > >> cranking them out?
> > > >> This is the first open source project I am looking to contribute to
> so
> > > >> forgive me If I am asking stupid questions.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <
> ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I also believe that grok parsers can be added through configuration
> > > only,
> > > >>> without having to
> > > >>> compile a parser.
> > > >>>
> > > >>> You can add a parser configuration targeting the basic grok parser
> > and
> > > >> just
> > > >>> provide the grok
> > > >>> parser rules.
> > > >>>
> > > >>>
> > > >>> Just as a heads up, I’m currently working on the parsers to allow
> for
> > > >>> writing and maintaining parsers
> > > >>> outside the metron code tree, including providing a maven
> archetype.
> > > >> This
> > > >>> will allow you to create parsers
> > > >>> without having to maintain a fork etc.
> > > >>>
> > > >>> Keep an eye out for METRON-258 as a PR on the list.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> > > wrote:
> > > >>>
> > > >>> My understanding of Grok vs Java is to provide a tradeoff for ease
> of
> > > >>> implementation vs performance (plus Java can also handle parsing
> that
> > > >> would
> > > >>> be too complicated for Grok.
> > > >>>
> > > >>> Grok is less performant and handles less complex parsing, but it's
> > easy
> > > >> to
> > > >>> get things going and potentially maintained without writing and
> > > compiling
> > > >>> Java.
> > > >>>
> > > >>> The Java implementation will be better for performance and can
> handle
> > > >> more
> > > >>> complicated parsing Grok can't.
> > > >>>
> > > >>> I believe the preference has generally been for Grok parsers if
> > > >>> appropriate, otherwise Java parsers.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
> alinazemian@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Mark,
> > > >>>>
> > > >>>> Yeah, that would be great. Can you please specify which devices
> you
> > > >> have
> > > >>>> developed so far?
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Ali
> > > >>>>
> > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.derijk@gmail.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Dear all,
> > > >>>>>
> > > >>>>> I am a heavy arcsight user and I have written quite a few parsers
> > > >> over
> > > >>>>> time.
> > > >>>>> I am new to contributing to open source projects however.
> > > >>>>> @Ali, would you like to cooperate on development of some parsers?
> > > >>>>>
> > > >>>>> Kind Regards,
> > > >>>>> Mark de Rijk
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> We are going to develop some parsers and have some contribution
> to
> > > >>> the
> > > >>>>>> community as a start point. I was wondering what the reason is
> > > >> behind
> > > >>>>>> choosing Grok statements for some of the implementations and
> Java
> > > >>> regex
> > > >>>>> for
> > > >>>>>> other ones? Is there any policy for that? Probably it would be
> > > >> better
> > > >>>> to
> > > >>>>>> have the Java regex implementation due to performance concerns.
> > > >>>> However,
> > > >>>>> I
> > > >>>>>> am sure there is a reason that some of them have been
> implemented
> > > >>> with
> > > >>>>>> using Grok statements.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Ali
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> A.Nazemian
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
Re: Develop some parsers
Posted by Nick Allen <ni...@nickallen.org>.
I think the primary limitation for parser contributors is that you need a
fair amount of good data to work with. Building a parser from a small
amount of test data will probably miss some corner cases that one would see
"in the wild." I don't think there are any right or wrong answers here
though. If you have access to good Infoblox data, for example, then start
there.
I would also suggest that you try building a parser with Grok expressions
first. You can do quite a bit with grok expressions. Get a feel for what
you can do with those, what the limitations are, and then transition into
Java parsers only as you see a need.
Really glad that you are joining the community. I look forward to your
contributions.
On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me...@gmail.com> wrote:
> Ali,
>
> Coding wise I am still getting my footing on and doing a Java course
> online.
> So building in unit tests I am afraid is a bit further away from me.
> For the parsers:
>
> - Infoblox Syslog
> - Cisco IOS Syslog
> - Unix Syslog
> - Alcatel Syslog
> - Sendmail Syslog
> - Blackberry Enterprise Server File
>
> I have also developed file and DB based connectors but as said I need to
> figure it out how to actually develop parsers for the Metron platform
> specifically.
>
> Is there a list of most requested devices documented somewhere so I can
> focus my efforts?
>
> Cheers,
> Mark
>
>
> On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Mark,
> >
> > Can you please specify the Parsers you are familiar with? We are
> > prioritizing the parser implementation, so your effort can help us to
> > target them faster.
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > I will review them this afternoon. Thanks for that.
> > >
> > > Sent from my iPhone
> > >
> > > > On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com> wrote:
> > > >
> > > > Mark,
> > > >
> > > > Have you seen the following pages?
> > > >
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > Development+Guidelines
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Metron+Development+
> > > Environment+Setup+Instructions
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Community+Resources
> > > >
> > > >
> > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> > > wrote:
> > > >>
> > > >> To clarify I have written a lot of parsers for ArcSight over the
> years
> > > and
> > > >> I would like to start contributing by developing parsers for the
> > Metron
> > > >> project.
> > > >> Is there any documentation that will help me get started so I can
> > start
> > > >> cranking them out?
> > > >> This is the first open source project I am looking to contribute to
> so
> > > >> forgive me If I am asking stupid questions.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <
> ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I also believe that grok parsers can be added through configuration
> > > only,
> > > >>> without having to
> > > >>> compile a parser.
> > > >>>
> > > >>> You can add a parser configuration targeting the basic grok parser
> > and
> > > >> just
> > > >>> provide the grok
> > > >>> parser rules.
> > > >>>
> > > >>>
> > > >>> Just as a heads up, I’m currently working on the parsers to allow
> for
> > > >>> writing and maintaining parsers
> > > >>> outside the metron code tree, including providing a maven
> archetype.
> > > >> This
> > > >>> will allow you to create parsers
> > > >>> without having to maintain a fork etc.
> > > >>>
> > > >>> Keep an eye out for METRON-258 as a PR on the list.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> > > wrote:
> > > >>>
> > > >>> My understanding of Grok vs Java is to provide a tradeoff for ease
> of
> > > >>> implementation vs performance (plus Java can also handle parsing
> that
> > > >> would
> > > >>> be too complicated for Grok.
> > > >>>
> > > >>> Grok is less performant and handles less complex parsing, but it's
> > easy
> > > >> to
> > > >>> get things going and potentially maintained without writing and
> > > compiling
> > > >>> Java.
> > > >>>
> > > >>> The Java implementation will be better for performance and can
> handle
> > > >> more
> > > >>> complicated parsing Grok can't.
> > > >>>
> > > >>> I believe the preference has generally been for Grok parsers if
> > > >>> appropriate, otherwise Java parsers.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
> alinazemian@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Mark,
> > > >>>>
> > > >>>> Yeah, that would be great. Can you please specify which devices
> you
> > > >> have
> > > >>>> developed so far?
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Ali
> > > >>>>
> > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.derijk@gmail.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Dear all,
> > > >>>>>
> > > >>>>> I am a heavy arcsight user and I have written quite a few parsers
> > > >> over
> > > >>>>> time.
> > > >>>>> I am new to contributing to open source projects however.
> > > >>>>> @Ali, would you like to cooperate on development of some parsers?
> > > >>>>>
> > > >>>>> Kind Regards,
> > > >>>>> Mark de Rijk
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> We are going to develop some parsers and have some contribution
> to
> > > >>> the
> > > >>>>>> community as a start point. I was wondering what the reason is
> > > >> behind
> > > >>>>>> choosing Grok statements for some of the implementations and
> Java
> > > >>> regex
> > > >>>>> for
> > > >>>>>> other ones? Is there any policy for that? Probably it would be
> > > >> better
> > > >>>> to
> > > >>>>>> have the Java regex implementation due to performance concerns.
> > > >>>> However,
> > > >>>>> I
> > > >>>>>> am sure there is a reason that some of them have been
> implemented
> > > >>> with
> > > >>>>>> using Grok statements.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Ali
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> A.Nazemian
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
Re: Develop some parsers
Posted by Mark de Rijk <me...@gmail.com>.
Ali,
Coding wise I am still getting my footing on and doing a Java course online.
So building in unit tests I am afraid is a bit further away from me.
For the parsers:
- Infoblox Syslog
- Cisco IOS Syslog
- Unix Syslog
- Alcatel Syslog
- Sendmail Syslog
- Blackberry Enterprise Server File
I have also developed file and DB based connectors but as said I need to
figure it out how to actually develop parsers for the Metron platform
specifically.
Is there a list of most requested devices documented somewhere so I can
focus my efforts?
Cheers,
Mark
On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <al...@gmail.com> wrote:
> Mark,
>
> Can you please specify the Parsers you are familiar with? We are
> prioritizing the parser implementation, so your effort can help us to
> target them faster.
>
> Cheers,
> Ali
>
> On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me...@gmail.com> wrote:
>
> > I will review them this afternoon. Thanks for that.
> >
> > Sent from my iPhone
> >
> > > On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com> wrote:
> > >
> > > Mark,
> > >
> > > Have you seen the following pages?
> > >
> > > https://cwiki.apache.org/confluence/display/METRON/
> > Development+Guidelines
> > > https://cwiki.apache.org/confluence/display/METRON/Metron+Development+
> > Environment+Setup+Instructions
> > > https://cwiki.apache.org/confluence/display/METRON/Community+Resources
> > >
> > >
> > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> > wrote:
> > >>
> > >> To clarify I have written a lot of parsers for ArcSight over the years
> > and
> > >> I would like to start contributing by developing parsers for the
> Metron
> > >> project.
> > >> Is there any documentation that will help me get started so I can
> start
> > >> cranking them out?
> > >> This is the first open source project I am looking to contribute to so
> > >> forgive me If I am asking stupid questions.
> > >>
> > >>
> > >>
> > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ot...@gmail.com>
> > >> wrote:
> > >>
> > >>> I also believe that grok parsers can be added through configuration
> > only,
> > >>> without having to
> > >>> compile a parser.
> > >>>
> > >>> You can add a parser configuration targeting the basic grok parser
> and
> > >> just
> > >>> provide the grok
> > >>> parser rules.
> > >>>
> > >>>
> > >>> Just as a heads up, I’m currently working on the parsers to allow for
> > >>> writing and maintaining parsers
> > >>> outside the metron code tree, including providing a maven archetype.
> > >> This
> > >>> will allow you to create parsers
> > >>> without having to maintain a fork etc.
> > >>>
> > >>> Keep an eye out for METRON-258 as a PR on the list.
> > >>>
> > >>>
> > >>>
> > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> > wrote:
> > >>>
> > >>> My understanding of Grok vs Java is to provide a tradeoff for ease of
> > >>> implementation vs performance (plus Java can also handle parsing that
> > >> would
> > >>> be too complicated for Grok.
> > >>>
> > >>> Grok is less performant and handles less complex parsing, but it's
> easy
> > >> to
> > >>> get things going and potentially maintained without writing and
> > compiling
> > >>> Java.
> > >>>
> > >>> The Java implementation will be better for performance and can handle
> > >> more
> > >>> complicated parsing Grok can't.
> > >>>
> > >>> I believe the preference has generally been for Grok parsers if
> > >>> appropriate, otherwise Java parsers.
> > >>>
> > >>> Justin
> > >>>
> > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Mark,
> > >>>>
> > >>>> Yeah, that would be great. Can you please specify which devices you
> > >> have
> > >>>> developed so far?
> > >>>>
> > >>>> Cheers,
> > >>>> Ali
> > >>>>
> > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> > >>> wrote:
> > >>>>
> > >>>>> Dear all,
> > >>>>>
> > >>>>> I am a heavy arcsight user and I have written quite a few parsers
> > >> over
> > >>>>> time.
> > >>>>> I am new to contributing to open source projects however.
> > >>>>> @Ali, would you like to cooperate on development of some parsers?
> > >>>>>
> > >>>>> Kind Regards,
> > >>>>> Mark de Rijk
> > >>>>>
> > >>>>>
> > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> > >> wrote:
> > >>>>>>
> > >>>>>> Hi all,
> > >>>>>>
> > >>>>>> We are going to develop some parsers and have some contribution to
> > >>> the
> > >>>>>> community as a start point. I was wondering what the reason is
> > >> behind
> > >>>>>> choosing Grok statements for some of the implementations and Java
> > >>> regex
> > >>>>> for
> > >>>>>> other ones? Is there any policy for that? Probably it would be
> > >> better
> > >>>> to
> > >>>>>> have the Java regex implementation due to performance concerns.
> > >>>> However,
> > >>>>> I
> > >>>>>> am sure there is a reason that some of them have been implemented
> > >>> with
> > >>>>>> using Grok statements.
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> Ali
> > >>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> A.Nazemian
> > >>>>
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > A.Nazemian
> >
>
>
>
> --
> A.Nazemian
>
Re: Develop some parsers
Posted by Ali Nazemian <al...@gmail.com>.
Mark,
Can you please specify the Parsers you are familiar with? We are
prioritizing the parser implementation, so your effort can help us to
target them faster.
Cheers,
Ali
On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me...@gmail.com> wrote:
> I will review them this afternoon. Thanks for that.
>
> Sent from my iPhone
>
> > On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com> wrote:
> >
> > Mark,
> >
> > Have you seen the following pages?
> >
> > https://cwiki.apache.org/confluence/display/METRON/
> Development+Guidelines
> > https://cwiki.apache.org/confluence/display/METRON/Metron+Development+
> Environment+Setup+Instructions
> > https://cwiki.apache.org/confluence/display/METRON/Community+Resources
> >
> >
> >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com>
> wrote:
> >>
> >> To clarify I have written a lot of parsers for ArcSight over the years
> and
> >> I would like to start contributing by developing parsers for the Metron
> >> project.
> >> Is there any documentation that will help me get started so I can start
> >> cranking them out?
> >> This is the first open source project I am looking to contribute to so
> >> forgive me If I am asking stupid questions.
> >>
> >>
> >>
> >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ot...@gmail.com>
> >> wrote:
> >>
> >>> I also believe that grok parsers can be added through configuration
> only,
> >>> without having to
> >>> compile a parser.
> >>>
> >>> You can add a parser configuration targeting the basic grok parser and
> >> just
> >>> provide the grok
> >>> parser rules.
> >>>
> >>>
> >>> Just as a heads up, I’m currently working on the parsers to allow for
> >>> writing and maintaining parsers
> >>> outside the metron code tree, including providing a maven archetype.
> >> This
> >>> will allow you to create parsers
> >>> without having to maintain a fork etc.
> >>>
> >>> Keep an eye out for METRON-258 as a PR on the list.
> >>>
> >>>
> >>>
> >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> wrote:
> >>>
> >>> My understanding of Grok vs Java is to provide a tradeoff for ease of
> >>> implementation vs performance (plus Java can also handle parsing that
> >> would
> >>> be too complicated for Grok.
> >>>
> >>> Grok is less performant and handles less complex parsing, but it's easy
> >> to
> >>> get things going and potentially maintained without writing and
> compiling
> >>> Java.
> >>>
> >>> The Java implementation will be better for performance and can handle
> >> more
> >>> complicated parsing Grok can't.
> >>>
> >>> I believe the preference has generally been for Grok parsers if
> >>> appropriate, otherwise Java parsers.
> >>>
> >>> Justin
> >>>
> >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Mark,
> >>>>
> >>>> Yeah, that would be great. Can you please specify which devices you
> >> have
> >>>> developed so far?
> >>>>
> >>>> Cheers,
> >>>> Ali
> >>>>
> >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Dear all,
> >>>>>
> >>>>> I am a heavy arcsight user and I have written quite a few parsers
> >> over
> >>>>> time.
> >>>>> I am new to contributing to open source projects however.
> >>>>> @Ali, would you like to cooperate on development of some parsers?
> >>>>>
> >>>>> Kind Regards,
> >>>>> Mark de Rijk
> >>>>>
> >>>>>
> >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> We are going to develop some parsers and have some contribution to
> >>> the
> >>>>>> community as a start point. I was wondering what the reason is
> >> behind
> >>>>>> choosing Grok statements for some of the implementations and Java
> >>> regex
> >>>>> for
> >>>>>> other ones? Is there any policy for that? Probably it would be
> >> better
> >>>> to
> >>>>>> have the Java regex implementation due to performance concerns.
> >>>> However,
> >>>>> I
> >>>>>> am sure there is a reason that some of them have been implemented
> >>> with
> >>>>>> using Grok statements.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Ali
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> A.Nazemian
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > A.Nazemian
>
--
A.Nazemian
Re: Develop some parsers
Posted by Mark De Rijk <me...@gmail.com>.
I will review them this afternoon. Thanks for that.
Sent from my iPhone
> On 7 Apr 2017, at 14:37, Ali Nazemian <al...@gmail.com> wrote:
>
> Mark,
>
> Have you seen the following pages?
>
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
>
>
>> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com> wrote:
>>
>> To clarify I have written a lot of parsers for ArcSight over the years and
>> I would like to start contributing by developing parsers for the Metron
>> project.
>> Is there any documentation that will help me get started so I can start
>> cranking them out?
>> This is the first open source project I am looking to contribute to so
>> forgive me If I am asking stupid questions.
>>
>>
>>
>> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> I also believe that grok parsers can be added through configuration only,
>>> without having to
>>> compile a parser.
>>>
>>> You can add a parser configuration targeting the basic grok parser and
>> just
>>> provide the grok
>>> parser rules.
>>>
>>>
>>> Just as a heads up, I’m currently working on the parsers to allow for
>>> writing and maintaining parsers
>>> outside the metron code tree, including providing a maven archetype.
>> This
>>> will allow you to create parsers
>>> without having to maintain a fork etc.
>>>
>>> Keep an eye out for METRON-258 as a PR on the list.
>>>
>>>
>>>
>>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
>>>
>>> My understanding of Grok vs Java is to provide a tradeoff for ease of
>>> implementation vs performance (plus Java can also handle parsing that
>> would
>>> be too complicated for Grok.
>>>
>>> Grok is less performant and handles less complex parsing, but it's easy
>> to
>>> get things going and potentially maintained without writing and compiling
>>> Java.
>>>
>>> The Java implementation will be better for performance and can handle
>> more
>>> complicated parsing Grok can't.
>>>
>>> I believe the preference has generally been for Grok parsers if
>>> appropriate, otherwise Java parsers.
>>>
>>> Justin
>>>
>>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
>>> wrote:
>>>
>>>> Hi Mark,
>>>>
>>>> Yeah, that would be great. Can you please specify which devices you
>> have
>>>> developed so far?
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
>>> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I am a heavy arcsight user and I have written quite a few parsers
>> over
>>>>> time.
>>>>> I am new to contributing to open source projects however.
>>>>> @Ali, would you like to cooperate on development of some parsers?
>>>>>
>>>>> Kind Regards,
>>>>> Mark de Rijk
>>>>>
>>>>>
>>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
>> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We are going to develop some parsers and have some contribution to
>>> the
>>>>>> community as a start point. I was wondering what the reason is
>> behind
>>>>>> choosing Grok statements for some of the implementations and Java
>>> regex
>>>>> for
>>>>>> other ones? Is there any policy for that? Probably it would be
>> better
>>>> to
>>>>>> have the Java regex implementation due to performance concerns.
>>>> However,
>>>>> I
>>>>>> am sure there is a reason that some of them have been implemented
>>> with
>>>>>> using Grok statements.
>>>>>>
>>>>>> Regards,
>>>>>> Ali
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> A.Nazemian
>>>>
>>>
>>
>
>
>
> --
> A.Nazemian
Re: Develop some parsers
Posted by Ali Nazemian <al...@gmail.com>.
Mark,
Have you seen the following pages?
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions
https://cwiki.apache.org/confluence/display/METRON/Community+Resources
On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me...@gmail.com> wrote:
> To clarify I have written a lot of parsers for ArcSight over the years and
> I would like to start contributing by developing parsers for the Metron
> project.
> Is there any documentation that will help me get started so I can start
> cranking them out?
> This is the first open source project I am looking to contribute to so
> forgive me If I am asking stupid questions.
>
>
>
> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > I also believe that grok parsers can be added through configuration only,
> > without having to
> > compile a parser.
> >
> > You can add a parser configuration targeting the basic grok parser and
> just
> > provide the grok
> > parser rules.
> >
> >
> > Just as a heads up, I’m currently working on the parsers to allow for
> > writing and maintaining parsers
> > outside the metron code tree, including providing a maven archetype.
> This
> > will allow you to create parsers
> > without having to maintain a fork etc.
> >
> > Keep an eye out for METRON-258 as a PR on the list.
> >
> >
> >
> > On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
> >
> > My understanding of Grok vs Java is to provide a tradeoff for ease of
> > implementation vs performance (plus Java can also handle parsing that
> would
> > be too complicated for Grok.
> >
> > Grok is less performant and handles less complex parsing, but it's easy
> to
> > get things going and potentially maintained without writing and compiling
> > Java.
> >
> > The Java implementation will be better for performance and can handle
> more
> > complicated parsing Grok can't.
> >
> > I believe the preference has generally been for Grok parsers if
> > appropriate, otherwise Java parsers.
> >
> > Justin
> >
> > On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> > wrote:
> >
> > > Hi Mark,
> > >
> > > Yeah, that would be great. Can you please specify which devices you
> have
> > > developed so far?
> > >
> > > Cheers,
> > > Ali
> > >
> > > On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> > wrote:
> > >
> > > > Dear all,
> > > >
> > > > I am a heavy arcsight user and I have written quite a few parsers
> over
> > > > time.
> > > > I am new to contributing to open source projects however.
> > > > @Ali, would you like to cooperate on development of some parsers?
> > > >
> > > > Kind Regards,
> > > > Mark de Rijk
> > > >
> > > >
> > > > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > We are going to develop some parsers and have some contribution to
> > the
> > > > > community as a start point. I was wondering what the reason is
> behind
> > > > > choosing Grok statements for some of the implementations and Java
> > regex
> > > > for
> > > > > other ones? Is there any policy for that? Probably it would be
> better
> > > to
> > > > > have the Java regex implementation due to performance concerns.
> > > However,
> > > > I
> > > > > am sure there is a reason that some of them have been implemented
> > with
> > > > > using Grok statements.
> > > > >
> > > > > Regards,
> > > > > Ali
> > > >
> > >
> > >
> > >
> > > --
> > > A.Nazemian
> > >
> >
>
--
A.Nazemian
Re: Develop some parsers
Posted by Mark de Rijk <me...@gmail.com>.
To clarify I have written a lot of parsers for ArcSight over the years and
I would like to start contributing by developing parsers for the Metron
project.
Is there any documentation that will help me get started so I can start
cranking them out?
This is the first open source project I am looking to contribute to so
forgive me If I am asking stupid questions.
On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <ot...@gmail.com> wrote:
> I also believe that grok parsers can be added through configuration only,
> without having to
> compile a parser.
>
> You can add a parser configuration targeting the basic grok parser and just
> provide the grok
> parser rules.
>
>
> Just as a heads up, I’m currently working on the parsers to allow for
> writing and maintaining parsers
> outside the metron code tree, including providing a maven archetype. This
> will allow you to create parsers
> without having to maintain a fork etc.
>
> Keep an eye out for METRON-258 as a PR on the list.
>
>
>
> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
>
> My understanding of Grok vs Java is to provide a tradeoff for ease of
> implementation vs performance (plus Java can also handle parsing that would
> be too complicated for Grok.
>
> Grok is less performant and handles less complex parsing, but it's easy to
> get things going and potentially maintained without writing and compiling
> Java.
>
> The Java implementation will be better for performance and can handle more
> complicated parsing Grok can't.
>
> I believe the preference has generally been for Grok parsers if
> appropriate, otherwise Java parsers.
>
> Justin
>
> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Hi Mark,
> >
> > Yeah, that would be great. Can you please specify which devices you have
> > developed so far?
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > Dear all,
> > >
> > > I am a heavy arcsight user and I have written quite a few parsers over
> > > time.
> > > I am new to contributing to open source projects however.
> > > @Ali, would you like to cooperate on development of some parsers?
> > >
> > > Kind Regards,
> > > Mark de Rijk
> > >
> > >
> > > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > We are going to develop some parsers and have some contribution to
> the
> > > > community as a start point. I was wondering what the reason is behind
> > > > choosing Grok statements for some of the implementations and Java
> regex
> > > for
> > > > other ones? Is there any policy for that? Probably it would be better
> > to
> > > > have the Java regex implementation due to performance concerns.
> > However,
> > > I
> > > > am sure there is a reason that some of them have been implemented
> with
> > > > using Grok statements.
> > > >
> > > > Regards,
> > > > Ali
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
Re: Develop some parsers
Posted by Otto Fowler <ot...@gmail.com>.
You’ll be able to use the archetype, even if you are doing configuration
only.
So - imagine this -
You want to do a grok ‘config’ only parser.
You :
* create a new parser extension with the archetype
* remove all the main/ java code
* setup your configuration in main/config ( for enrichments, indexing, and
parser )
* you add your grok patterns
* you can write unit and integration tests against your grok parser rules
etc
* you package it all up
* you deploy as a regular extension
You *could* do just a configuration, but I think being able to write unit
tests etc is better ;)
On April 7, 2017 at 09:35:20, Ali Nazemian (alinazemian@gmail.com) wrote:
Having a separate maven archetype is really great.
I prefer to use the Java one because performance is really important for
us...
On Fri, Apr 7, 2017 at 11:13 PM, Otto Fowler <ot...@gmail.com>
wrote:
> I also believe that grok parsers can be added through configuration only,
> without having to
> compile a parser.
>
> You can add a parser configuration targeting the basic grok parser and
just
> provide the grok
> parser rules.
>
>
> Just as a heads up, I’m currently working on the parsers to allow for
> writing and maintaining parsers
> outside the metron code tree, including providing a maven archetype. This
> will allow you to create parsers
> without having to maintain a fork etc.
>
> Keep an eye out for METRON-258 as a PR on the list.
>
>
>
> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
>
> My understanding of Grok vs Java is to provide a tradeoff for ease of
> implementation vs performance (plus Java can also handle parsing that
would
> be too complicated for Grok.
>
> Grok is less performant and handles less complex parsing, but it's easy
to
> get things going and potentially maintained without writing and compiling
> Java.
>
> The Java implementation will be better for performance and can handle
more
> complicated parsing Grok can't.
>
> I believe the preference has generally been for Grok parsers if
> appropriate, otherwise Java parsers.
>
> Justin
>
> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Hi Mark,
> >
> > Yeah, that would be great. Can you please specify which devices you
have
> > developed so far?
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > Dear all,
> > >
> > > I am a heavy arcsight user and I have written quite a few parsers
over
> > > time.
> > > I am new to contributing to open source projects however.
> > > @Ali, would you like to cooperate on development of some parsers?
> > >
> > > Kind Regards,
> > > Mark de Rijk
> > >
> > >
> > > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com>
wrote:
> > > >
> > > > Hi all,
> > > >
> > > > We are going to develop some parsers and have some contribution to
> the
> > > > community as a start point. I was wondering what the reason is
behind
> > > > choosing Grok statements for some of the implementations and Java
> regex
> > > for
> > > > other ones? Is there any policy for that? Probably it would be
better
> > to
> > > > have the Java regex implementation due to performance concerns.
> > However,
> > > I
> > > > am sure there is a reason that some of them have been implemented
> with
> > > > using Grok statements.
> > > >
> > > > Regards,
> > > > Ali
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
--
A.Nazemian
Re: Develop some parsers
Posted by Ali Nazemian <al...@gmail.com>.
Having a separate maven archetype is really great.
I prefer to use the Java one because performance is really important for
us...
On Fri, Apr 7, 2017 at 11:13 PM, Otto Fowler <ot...@gmail.com>
wrote:
> I also believe that grok parsers can be added through configuration only,
> without having to
> compile a parser.
>
> You can add a parser configuration targeting the basic grok parser and just
> provide the grok
> parser rules.
>
>
> Just as a heads up, I’m currently working on the parsers to allow for
> writing and maintaining parsers
> outside the metron code tree, including providing a maven archetype. This
> will allow you to create parsers
> without having to maintain a fork etc.
>
> Keep an eye out for METRON-258 as a PR on the list.
>
>
>
> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
>
> My understanding of Grok vs Java is to provide a tradeoff for ease of
> implementation vs performance (plus Java can also handle parsing that would
> be too complicated for Grok.
>
> Grok is less performant and handles less complex parsing, but it's easy to
> get things going and potentially maintained without writing and compiling
> Java.
>
> The Java implementation will be better for performance and can handle more
> complicated parsing Grok can't.
>
> I believe the preference has generally been for Grok parsers if
> appropriate, otherwise Java parsers.
>
> Justin
>
> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Hi Mark,
> >
> > Yeah, that would be great. Can you please specify which devices you have
> > developed so far?
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com>
> wrote:
> >
> > > Dear all,
> > >
> > > I am a heavy arcsight user and I have written quite a few parsers over
> > > time.
> > > I am new to contributing to open source projects however.
> > > @Ali, would you like to cooperate on development of some parsers?
> > >
> > > Kind Regards,
> > > Mark de Rijk
> > >
> > >
> > > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > We are going to develop some parsers and have some contribution to
> the
> > > > community as a start point. I was wondering what the reason is behind
> > > > choosing Grok statements for some of the implementations and Java
> regex
> > > for
> > > > other ones? Is there any policy for that? Probably it would be better
> > to
> > > > have the Java regex implementation due to performance concerns.
> > However,
> > > I
> > > > am sure there is a reason that some of them have been implemented
> with
> > > > using Grok statements.
> > > >
> > > > Regards,
> > > > Ali
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
--
A.Nazemian
Re: Develop some parsers
Posted by Otto Fowler <ot...@gmail.com>.
I also believe that grok parsers can be added through configuration only,
without having to
compile a parser.
You can add a parser configuration targeting the basic grok parser and just
provide the grok
parser rules.
Just as a heads up, I’m currently working on the parsers to allow for
writing and maintaining parsers
outside the metron code tree, including providing a maven archetype. This
will allow you to create parsers
without having to maintain a fork etc.
Keep an eye out for METRON-258 as a PR on the list.
On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com) wrote:
My understanding of Grok vs Java is to provide a tradeoff for ease of
implementation vs performance (plus Java can also handle parsing that would
be too complicated for Grok.
Grok is less performant and handles less complex parsing, but it's easy to
get things going and potentially maintained without writing and compiling
Java.
The Java implementation will be better for performance and can handle more
complicated parsing Grok can't.
I believe the preference has generally been for Grok parsers if
appropriate, otherwise Java parsers.
Justin
On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com> wrote:
> Hi Mark,
>
> Yeah, that would be great. Can you please specify which devices you have
> developed so far?
>
> Cheers,
> Ali
>
> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com> wrote:
>
> > Dear all,
> >
> > I am a heavy arcsight user and I have written quite a few parsers over
> > time.
> > I am new to contributing to open source projects however.
> > @Ali, would you like to cooperate on development of some parsers?
> >
> > Kind Regards,
> > Mark de Rijk
> >
> >
> > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > We are going to develop some parsers and have some contribution to
the
> > > community as a start point. I was wondering what the reason is behind
> > > choosing Grok statements for some of the implementations and Java
regex
> > for
> > > other ones? Is there any policy for that? Probably it would be better
> to
> > > have the Java regex implementation due to performance concerns.
> However,
> > I
> > > am sure there is a reason that some of them have been implemented
with
> > > using Grok statements.
> > >
> > > Regards,
> > > Ali
> >
>
>
>
> --
> A.Nazemian
>
Re: Develop some parsers
Posted by Justin Leet <ju...@gmail.com>.
My understanding of Grok vs Java is to provide a tradeoff for ease of
implementation vs performance (plus Java can also handle parsing that would
be too complicated for Grok.
Grok is less performant and handles less complex parsing, but it's easy to
get things going and potentially maintained without writing and compiling
Java.
The Java implementation will be better for performance and can handle more
complicated parsing Grok can't.
I believe the preference has generally been for Grok parsers if
appropriate, otherwise Java parsers.
Justin
On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <al...@gmail.com> wrote:
> Hi Mark,
>
> Yeah, that would be great. Can you please specify which devices you have
> developed so far?
>
> Cheers,
> Ali
>
> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com> wrote:
>
> > Dear all,
> >
> > I am a heavy arcsight user and I have written quite a few parsers over
> > time.
> > I am new to contributing to open source projects however.
> > @Ali, would you like to cooperate on development of some parsers?
> >
> > Kind Regards,
> > Mark de Rijk
> >
> >
> > > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > We are going to develop some parsers and have some contribution to the
> > > community as a start point. I was wondering what the reason is behind
> > > choosing Grok statements for some of the implementations and Java regex
> > for
> > > other ones? Is there any policy for that? Probably it would be better
> to
> > > have the Java regex implementation due to performance concerns.
> However,
> > I
> > > am sure there is a reason that some of them have been implemented with
> > > using Grok statements.
> > >
> > > Regards,
> > > Ali
> >
>
>
>
> --
> A.Nazemian
>
Re: Develop some parsers
Posted by Ali Nazemian <al...@gmail.com>.
Hi Mark,
Yeah, that would be great. Can you please specify which devices you have
developed so far?
Cheers,
Ali
On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me...@gmail.com> wrote:
> Dear all,
>
> I am a heavy arcsight user and I have written quite a few parsers over
> time.
> I am new to contributing to open source projects however.
> @Ali, would you like to cooperate on development of some parsers?
>
> Kind Regards,
> Mark de Rijk
>
>
> > On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com> wrote:
> >
> > Hi all,
> >
> > We are going to develop some parsers and have some contribution to the
> > community as a start point. I was wondering what the reason is behind
> > choosing Grok statements for some of the implementations and Java regex
> for
> > other ones? Is there any policy for that? Probably it would be better to
> > have the Java regex implementation due to performance concerns. However,
> I
> > am sure there is a reason that some of them have been implemented with
> > using Grok statements.
> >
> > Regards,
> > Ali
>
--
A.Nazemian
Re: Develop some parsers
Posted by Mark De Rijk <me...@gmail.com>.
Dear all,
I am a heavy arcsight user and I have written quite a few parsers over time.
I am new to contributing to open source projects however.
@Ali, would you like to cooperate on development of some parsers?
Kind Regards,
Mark de Rijk
> On 7 Apr 2017, at 04:30, Ali Nazemian <al...@gmail.com> wrote:
>
> Hi all,
>
> We are going to develop some parsers and have some contribution to the
> community as a start point. I was wondering what the reason is behind
> choosing Grok statements for some of the implementations and Java regex for
> other ones? Is there any policy for that? Probably it would be better to
> have the Java regex implementation due to performance concerns. However, I
> am sure there is a reason that some of them have been implemented with
> using Grok statements.
>
> Regards,
> Ali