You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Simon Elliston Ball <si...@simonellistonball.com> on 2017/05/08 23:35:31 UTC
Parser Docs
Quick thought, and please shoot me down if this has already been thought of, but….
Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example.
Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!)
Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with.
Simon
Re: Parser Docs
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Hi Mark,
If you’re looking to write Metron parsers you aren’t going to have to worry about Nifi or Kafka in any level of detail. The parser interface just gets byte[] and outputs JSON.
Of course I would never recommend avoiding the reading around all the other exciting bits and components around the eco-system, but if you want to dive in sooner, Metron parsers are a nice easy and self-contained place to start.
Simon
> On 11 May 2017, at 14:38, Mark de Rijk <ma...@samarkconsulting.co.uk> wrote:
>
> Hi,
>
> I really would like to help with parser development. I am reading up on Nifi, Kafka and there are probably other topics I am missing now.
> As when it comes to Hadoop ecosystem I am pretty much a beginner I will unfortunately need a month to learn the different components.
>
> Regards,
> Mark de Rijk
>
>
>
> Sent from my iPhone
>
>> On 9 May 2017, at 02:23, Otto Fowler <ot...@gmail.com> wrote:
>>
>> There is a readme.md PER parser in 777.
>> I only stubbed them out however.
>>
>> Each parser created by the archetype has one as well.
>>
>> What I was hoping to do was to include the parser docs in the package assembly so the UI could load it.
>>
>>
>>
>> On May 8, 2017 at 19:35:41, Simon Elliston Ball (simon@simonellistonball.com) wrote:
>>
>> Quick thought, and please shoot me down if this has already been thought of, but….
>>
>> Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example.
>>
>> Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!)
>>
>> Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with.
>>
>> Simon
Re: Parser Docs
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Well, you do at the moment… I’m still keen on the idea of parsers emitting a schema of sorts, and then the framework being expanded to use that schema, along with enrichment schema and stellar output type inference to generate ES templates, but that’s another story. ES Template specs certainly do not belong in parsers, because they have to account for Enrichment, Intel and Triage data as well, which are not known to the parser stage.
Agreed though, right now a working knowledge of ES templates is useful for parser dev.
Simon
> On 11 May 2017, at 14:51, Otto Fowler <ot...@gmail.com> wrote:
>
> I missed elasticsearch, you do need to understand ES indexing to setup the correct storage etc for the fields you produce.
>
>
>
> On May 11, 2017 at 09:48:10, Otto Fowler (ottobackwards@gmail.com <ma...@gmail.com>) wrote:
>
>> Part of the point of having a framework like metron is that you don’t *need* to know those things to contribute parsers.
>> What you would want to understand are the things at parser scope:
>>
>> STELLAR,
>> the MessageParser interface
>> The base ‘typed’ parsers ( JSONMap, CSVParser, BasicGrokParser )
>>
>> and your data.
>>
>> Even the integration testing framework has been done in such a way that you can test with indexing, enrichment etc without having to worry
>> about setting up zookeeper, kafka, storm etc.
>>
>>
>> On May 11, 2017 at 09:38:46, Mark de Rijk (mark.derijk@samarkconsulting.co.uk <ma...@samarkconsulting.co.uk>) wrote:
>>
>>> Hi,
>>>
>>> I really would like to help with parser development. I am reading up on Nifi, Kafka and there are probably other topics I am missing now.
>>> As when it comes to Hadoop ecosystem I am pretty much a beginner I will unfortunately need a month to learn the different components.
>>>
>>> Regards,
>>> Mark de Rijk
>>>
>>>
>>>
>>> Sent from my iPhone
>>>
>>> > On 9 May 2017, at 02:23, Otto Fowler <ottobackwards@gmail.com <ma...@gmail.com>> wrote:
>>> >
>>> > There is a readme.md <http://readme.md/> PER parser in 777.
>>> > I only stubbed them out however.
>>> >
>>> > Each parser created by the archetype has one as well.
>>> >
>>> > What I was hoping to do was to include the parser docs in the package assembly so the UI could load it.
>>> >
>>> >
>>> >
>>> > On May 8, 2017 at 19:35:41, Simon Elliston Ball (simon@simonellistonball.com <ma...@simonellistonball.com>) wrote:
>>> >
>>> > Quick thought, and please shoot me down if this has already been thought of, but….
>>> >
>>> > Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example.
>>> >
>>> > Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!)
>>> >
>>> > Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with.
>>> >
>>> > Simon
Re: Parser Docs
Posted by Otto Fowler <ot...@gmail.com>.
I missed elasticsearch, you do need to understand ES indexing to setup the
correct storage etc for the fields you produce.
On May 11, 2017 at 09:48:10, Otto Fowler (ottobackwards@gmail.com) wrote:
Part of the point of having a framework like metron is that you don’t
*need* to know those things to contribute parsers.
What you would want to understand are the things at parser scope:
STELLAR,
the MessageParser interface
The base ‘typed’ parsers ( JSONMap, CSVParser, BasicGrokParser )
and your data.
Even the integration testing framework has been done in such a way that you
can test with indexing, enrichment etc without having to worry
about setting up zookeeper, kafka, storm etc.
On May 11, 2017 at 09:38:46, Mark de Rijk (
mark.derijk@samarkconsulting.co.uk) wrote:
Hi,
I really would like to help with parser development. I am reading up on
Nifi, Kafka and there are probably other topics I am missing now.
As when it comes to Hadoop ecosystem I am pretty much a beginner I will
unfortunately need a month to learn the different components.
Regards,
Mark de Rijk
Sent from my iPhone
> On 9 May 2017, at 02:23, Otto Fowler <ot...@gmail.com> wrote:
>
> There is a readme.md PER parser in 777.
> I only stubbed them out however.
>
> Each parser created by the archetype has one as well.
>
> What I was hoping to do was to include the parser docs in the package
assembly so the UI could load it.
>
>
>
> On May 8, 2017 at 19:35:41, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
>
> Quick thought, and please shoot me down if this has already been thought
of, but….
>
> Would it be worthwhile if we put some per parser docs into the repo,
essentially a README.md per major parser class, which would in theory be
picked up by the docbook? We don’t really have much documentation right now
on the effect of parserConfig settings for example.
>
> Similarly, is this something worth considering in the package format, or
at least in the archetype proposed by METRON-777 (paging Ottto!)
>
> Worth the effort? I’m happy to do at least a few of the more recent ones
I’ve spent meaningful time with.
>
> Simon
Re: Parser Docs
Posted by Otto Fowler <ot...@gmail.com>.
Part of the point of having a framework like metron is that you don’t
*need* to know those things to contribute parsers.
What you would want to understand are the things at parser scope:
STELLAR,
the MessageParser interface
The base ‘typed’ parsers ( JSONMap, CSVParser, BasicGrokParser )
and your data.
Even the integration testing framework has been done in such a way that you
can test with indexing, enrichment etc without having to worry
about setting up zookeeper, kafka, storm etc.
On May 11, 2017 at 09:38:46, Mark de Rijk (
mark.derijk@samarkconsulting.co.uk) wrote:
Hi,
I really would like to help with parser development. I am reading up on
Nifi, Kafka and there are probably other topics I am missing now.
As when it comes to Hadoop ecosystem I am pretty much a beginner I will
unfortunately need a month to learn the different components.
Regards,
Mark de Rijk
Sent from my iPhone
> On 9 May 2017, at 02:23, Otto Fowler <ot...@gmail.com> wrote:
>
> There is a readme.md PER parser in 777.
> I only stubbed them out however.
>
> Each parser created by the archetype has one as well.
>
> What I was hoping to do was to include the parser docs in the package
assembly so the UI could load it.
>
>
>
> On May 8, 2017 at 19:35:41, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
>
> Quick thought, and please shoot me down if this has already been thought
of, but….
>
> Would it be worthwhile if we put some per parser docs into the repo,
essentially a README.md per major parser class, which would in theory be
picked up by the docbook? We don’t really have much documentation right now
on the effect of parserConfig settings for example.
>
> Similarly, is this something worth considering in the package format, or
at least in the archetype proposed by METRON-777 (paging Ottto!)
>
> Worth the effort? I’m happy to do at least a few of the more recent ones
I’ve spent meaningful time with.
>
> Simon
Re: Parser Docs
Posted by Mark de Rijk <ma...@samarkconsulting.co.uk>.
Hi,
I really would like to help with parser development. I am reading up on Nifi, Kafka and there are probably other topics I am missing now.
As when it comes to Hadoop ecosystem I am pretty much a beginner I will unfortunately need a month to learn the different components.
Regards,
Mark de Rijk
Sent from my iPhone
> On 9 May 2017, at 02:23, Otto Fowler <ot...@gmail.com> wrote:
>
> There is a readme.md PER parser in 777.
> I only stubbed them out however.
>
> Each parser created by the archetype has one as well.
>
> What I was hoping to do was to include the parser docs in the package assembly so the UI could load it.
>
>
>
> On May 8, 2017 at 19:35:41, Simon Elliston Ball (simon@simonellistonball.com) wrote:
>
> Quick thought, and please shoot me down if this has already been thought of, but….
>
> Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example.
>
> Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!)
>
> Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with.
>
> Simon
Re: Parser Docs
Posted by Otto Fowler <ot...@gmail.com>.
Just to be clear, I did not include the docs in the assembly… there is a lot in 777 already ;)
On May 8, 2017 at 21:23:28, Otto Fowler (ottobackwards@gmail.com) wrote:
There is a readme.md PER parser in 777.
I only stubbed them out however.
Each parser created by the archetype has one as well.
What I was hoping to do was to include the parser docs in the package assembly so the UI could load it.
On May 8, 2017 at 19:35:41, Simon Elliston Ball (simon@simonellistonball.com) wrote:
Quick thought, and please shoot me down if this has already been thought of, but….
Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example.
Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!)
Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with.
Simon
Re: Parser Docs
Posted by Otto Fowler <ot...@gmail.com>.
There is a readme.md PER parser in 777.
I only stubbed them out however.
Each parser created by the archetype has one as well.
What I was hoping to do was to include the parser docs in the package assembly so the UI could load it.
On May 8, 2017 at 19:35:41, Simon Elliston Ball (simon@simonellistonball.com) wrote:
Quick thought, and please shoot me down if this has already been thought of, but….
Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example.
Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!)
Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with.
Simon
Re: Parser Docs
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Definitely worthwhile. I discussed something similar (but more general) a
little while back here
<https://lists.apache.org/thread.html/17904a10b8f41a97dfac84dcb3c22a1a0229fa9db170fa762741c2fe@%3Cdev.metron.apache.org%3E>.
Totally worth the effort IMO.
Jon
On Mon, May 8, 2017 at 7:36 PM Casey Stella <ce...@gmail.com> wrote:
> +1 for parser docs
>
> On Mon, May 8, 2017 at 7:35 PM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> > Quick thought, and please shoot me down if this has already been thought
> > of, but….
> >
> > Would it be worthwhile if we put some per parser docs into the repo,
> > essentially a README.md per major parser class, which would in theory be
> > picked up by the docbook? We don’t really have much documentation right
> now
> > on the effect of parserConfig settings for example.
> >
> > Similarly, is this something worth considering in the package format, or
> > at least in the archetype proposed by METRON-777 (paging Ottto!)
> >
> > Worth the effort? I’m happy to do at least a few of the more recent ones
> > I’ve spent meaningful time with.
> >
> > Simon
>
--
Jon
Re: Parser Docs
Posted by Casey Stella <ce...@gmail.com>.
+1 for parser docs
On Mon, May 8, 2017 at 7:35 PM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> Quick thought, and please shoot me down if this has already been thought
> of, but….
>
> Would it be worthwhile if we put some per parser docs into the repo,
> essentially a README.md per major parser class, which would in theory be
> picked up by the docbook? We don’t really have much documentation right now
> on the effect of parserConfig settings for example.
>
> Similarly, is this something worth considering in the package format, or
> at least in the archetype proposed by METRON-777 (paging Ottto!)
>
> Worth the effort? I’m happy to do at least a few of the more recent ones
> I’ve spent meaningful time with.
>
> Simon