You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2016/09/26 17:35:11 UTC
Complete steps to add a new parser
Are all the steps required to add a parser documented anywhere? The squid
document starts the topology, but I don’t think that integrates it in with
monit for example. Or does that actually happen?
Re: Complete steps to add a new parser
Posted by Otto Fowler <ot...@gmail.com>.
To clarify:
I would like to add a new parser to deployment, such that it will produce
yaf, bro, snort, myparser with full support into the system including monit
etc when I deploy to vagrant or to my cluster.
Is there a how to, commit, wiki entry for this?
On September 26, 2016 at 13:35:11, Otto Fowler (ottobackwards@gmail.com)
wrote:
Are all the steps required to add a parser documented anywhere? The squid
document starts the topology, but I don’t think that integrates it in with
monit for example. Or does that actually happen?
Re: Complete steps to add a new parser
Posted by Nick Allen <ni...@nickallen.org>.
A few other points of interest...
*Sample PCAP* - The sample pcap data that is replayed is installed by the
'sensor-test-mode' that lives at
`metron-deployment/roles/sensor-test-mode`. If you don't want to use that,
just don't turn off 'sensor-test-mode'.
*ES Templates* - These live under
`metron-deployments/roles/metron_elasticsearch_templates`. Imagine that. :)
On Tue, Sep 27, 2016 at 12:31 PM, Nick Allen <ni...@nickallen.org> wrote:
> Hi Otto -
>
> I would agree with you. We do not have documentation that describes how
> to 'permanently install' a new parser.. Your contribution would be highly
> appreciated in this area.
>
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code. An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser. But of course, if we can find
> ways to make this task easier, we will.
>
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
>
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of metron-deployment/roles/monit/templates/monit/parsers.monit
> to add your own parser definition to Monit.
>
> *Parsers* - The start script in `metron-platform/metron-
> parsers/src/main/scripts/start_parser_topology.sh` will give you good
> hooks into how each of the parsers are started.
>
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`. To
> understand that process, start at `tasks/main.yml`.
>
> I probably missed something, but let me know if you have questions.
>
>
>
>
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> My wish, is that when I do an ansible-playbook -v -i {my configuration}
>> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
>> my parser / topology is deployed, started and monitored the same way as
>> the
>> current bro, snort, and yaf parsers are.
>>
>> I might be misunderstanding something however. I seems to me that all the
>> examples of adding other parsers are temporary and not permanent because
>> they do not have the full deployment, kind of push the config and run the
>> script and you are going. Am I missing something? Would the squid sample
>> steps result in a parser topology that would survive restarts / reboots
>> etc?
>>
>> On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> Just so I completely understand what you are asking for...you want to know
>> how to create a new parser topology with the JSON parser and plug it into
>> Monit so you can monitor and restart it on demand?
>>
>> 27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
>> > Thanks James,
>> >
>> > I want to deploy an instance of the JSONMapParser into my POC cluster
>> and
>> vagrant. I’m trying to work out exactly how to add a new configured
>> parser
>> instance to the deployment. I think these instructions would be a good
>> extension to the squid stuff that is already there. If I could get that
>> going and add a new parser all the way through, then maybe I can
>> contribute
>> something in that area. The ability to do this will also enable some of
>> the other work you mentioned.
>> >
>> > On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
>> wrote:
>> >
>> >> There are three types of parsers you can have currently. Our preferred
>> way is to use Grok parser. The only thing you need to do there is to
>> define
>> your Grok statement and the parser will uptake it and do the rest. That is
>> what most of our documentation reflect. The second type of parser that we
>> have is a java parser, where you actually extend a parser class and define
>> your own custom parsing logic. We intend this type of parser for high
>> velocity feeds that require custom parsing logic that is not easily
>> attainable by Grok. The third type of parser is the one you have been
>> working on, a Json parser. This is a parser designed to take pre-parsed
>> JSON for sensors that either log in JSON format natively or have been
>> pre-parsed for us by some system upstream.
>> >>
>> >> Parsers don't integrate with Monit by default. We can come up with some
>> instructions for you on how to do that.
>> >>
>> >> I should also note there are 2 additional parser types that are on the
>> road map. METRON-295 (scripting bolt), which is a parser that allows you
>> to
>> uptake something like javascript, lua, etc., for doing the parsing. There
>> is also METRON-288, which is a XSL parser designed to parse XML documents.
>> If either of these are of interest to you we would welcome this
>> contribution and we can work with you to get you started.
>> >>
>> >> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
>> >>> Are all the steps required to add a parser documented anywhere? The
>> squid
>> >>> document starts the topology, but I don’t think that integrates it in
>> with
>> >>> monit for example. Or does that actually happen?
>> >>
>> >> -------------------
>> >> Thank you,
>> >>
>> >> James Sirota
>> >> PPMC- Apache Metron (Incubating)
>> >> jsirota AT apache DOT org
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
>>
>
>
>
> --
> Nick Allen <ni...@nickallen.org>
>
--
Nick Allen <ni...@nickallen.org>
Re: Complete steps to add a new parser
Posted by Nick Allen <ni...@nickallen.org>.
(Meant to send this to the mailing list earlier, but mistakenly replied
only to Otto.)
A while back I put together a repo that shows how you could maintain
settings for your Metron deployments outside of the Metron code base
itself. This may be useful to you or others who are trying to deploy
Metron with Ansible. It may be a little dated, but at least gives some
structure showing how this could be managed.
https://github.com/nickwallen/metron-environments
On Sat, Oct 1, 2016 at 11:59 PM, Otto Fowler <ot...@gmail.com>
wrote:
> I have been able to add a new parser to the the deployment, and have the
> cluster fully deploy successfully. After I was able to push data to kafka
> from HDF and get it all indexed.
>
> Unlike quick dev and full, no problems getting the storm ports correct
> after deployment to my small cluster config.
>
> It looks to me that the steps I took to integrate the parser worked, but I
> still may have missed something. One thing that I know I missed was
> modifying the dashboard - adding the saved searches and integrating them
> with the visualizations.
>
> Here is a gist off a patch of my changes. The patch in the gist has been
> modified - so I don’t think it will apply for you. I removed proprietary
> field names ( the stellar config, the enrichment hosts, es index template ).
>
> I hope what is there is enough for you to verify, correct what I have done.
>
> https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84
>
>
>
> On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> Thanks Nick,
> That is some of the stuff that I have found trying to track down the
> deploy bits of the existing parsers, but I don’t want to miss anything, so
> I’d like some guidence. Right now, I’m OK with doing it all inside the
> metron ansible base. I expect that once I get it working and wrap my head
> around it I’ll have some ideas that I’ll float around improving this area,
> some use cases to propose that would possibly be external to the main
> deployment, or additive. First thing first is understanding all the points
> in doing it the hard way ;)
>
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 12:41:10, Nick Allen (nick@nickallen.org) wrote:
>
> Hi Otto -
>
> I would agree with you. We do not have documentation that describes how to
> 'permanently install' a new parser.. Your contribution would be highly
> appreciated in this area.
>
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code. An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser. But of course, if we can find
> ways to make this task easier, we will.
>
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
>
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of
> metron-deployment/roles/monit/templates/monit/parsers.monit to add your
> own
> parser definition to Monit.
>
> *Parsers* - The start script in
> `metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
> will give you good hooks into how each of the parsers are started.
>
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`. To understand
> that process, start at `tasks/main.yml`.
>
> I probably missed something, but let me know if you have questions.
>
>
>
>
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > My wish, is that when I do an ansible-playbook -v -i {my configuration}
> > metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> > my parser / topology is deployed, started and monitored the same way as
> the
> > current bro, snort, and yaf parsers are.
> >
> > I might be misunderstanding something however. I seems to me that all the
> > examples of adding other parsers are temporary and not permanent because
> > they do not have the full deployment, kind of push the config and run the
> > script and you are going. Am I missing something? Would the squid sample
> > steps result in a parser topology that would survive restarts / reboots
> > etc?
> >
> > On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
> > wrote:
> >
> > Just so I completely understand what you are asking for...you want to
> know
> > how to create a new parser topology with the JSON parser and plug it into
> > Monit so you can monitor and restart it on demand?
> >
> > 27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> > > Thanks James,
> > >
> > > I want to deploy an instance of the JSONMapParser into my POC cluster
> and
> > vagrant. I’m trying to work out exactly how to add a new configured
> parser
> > instance to the deployment. I think these instructions would be a good
> > extension to the squid stuff that is already there. If I could get that
> > going and add a new parser all the way through, then maybe I can
> contribute
> > something in that area. The ability to do this will also enable some of
> > the other work you mentioned.
> > >
> > > On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
> > wrote:
> > >
> > >> There are three types of parsers you can have currently. Our preferred
> > way is to use Grok parser. The only thing you need to do there is to
> define
> > your Grok statement and the parser will uptake it and do the rest. That
> is
> > what most of our documentation reflect. The second type of parser that we
> > have is a java parser, where you actually extend a parser class and
> define
> > your own custom parsing logic. We intend this type of parser for high
> > velocity feeds that require custom parsing logic that is not easily
> > attainable by Grok. The third type of parser is the one you have been
> > working on, a Json parser. This is a parser designed to take pre-parsed
> > JSON for sensors that either log in JSON format natively or have been
> > pre-parsed for us by some system upstream.
> > >>
> > >> Parsers don't integrate with Monit by default. We can come up with
> some
> > instructions for you on how to do that.
> > >>
> > >> I should also note there are 2 additional parser types that are on the
> > road map. METRON-295 (scripting bolt), which is a parser that allows you
> to
> > uptake something like javascript, lua, etc., for doing the parsing. There
> > is also METRON-288, which is a XSL parser designed to parse XML
> documents.
> > If either of these are of interest to you we would welcome this
> > contribution and we can work with you to get you started.
> > >>
> > >> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> > >>> Are all the steps required to add a parser documented anywhere? The
> > squid
> > >>> document starts the topology, but I don’t think that integrates it in
> > with
> > >>> monit for example. Or does that actually happen?
> > >>
> > >> -------------------
> > >> Thank you,
> > >>
> > >> James Sirota
> > >> PPMC- Apache Metron (Incubating)
> > >> jsirota AT apache DOT org
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
>
>
>
> --
> Nick Allen <ni...@nickallen.org>
>
>
--
Nick Allen <ni...@nickallen.org>
Re: Complete steps to add a new parser
Posted by James Sirota <js...@apache.org>.
Thanks for doing the work, Otto. We'll take a look
01.10.2016, 21:00, "Otto Fowler" <ot...@gmail.com>:
> I have been able to add a new parser to the the deployment, and have the
> cluster fully deploy successfully. After I was able to push data to kafka
> from HDF and get it all indexed.
>
> Unlike quick dev and full, no problems getting the storm ports correct
> after deployment to my small cluster config.
>
> It looks to me that the steps I took to integrate the parser worked, but I
> still may have missed something. One thing that I know I missed was
> modifying the dashboard - adding the saved searches and integrating them
> with the visualizations.
>
> Here is a gist off a patch of my changes. The patch in the gist has been
> modified - so I don\u2019t think it will apply for you. I removed proprietary
> field names ( the stellar config, the enrichment hosts, es index template ).
>
> I hope what is there is enough for you to verify, correct what I have done.
>
> https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84
>
> On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> Thanks Nick,
> That is some of the stuff that I have found trying to track down the deploy
> bits of the existing parsers, but I don\u2019t want to miss anything, so I\u2019d
> like some guidence. Right now, I\u2019m OK with doing it all inside the metron
> ansible base. I expect that once I get it working and wrap my head around
> it I\u2019ll have some ideas that I\u2019ll float around improving this area, some
> use cases to propose that would possibly be external to the main
> deployment, or additive. First thing first is understanding all the points
> in doing it the hard way ;)
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 12:41:10, Nick Allen (nick@nickallen.org) wrote:
>
> Hi Otto -
>
> I would agree with you. We do not have documentation that describes how to
> 'permanently install' a new parser.. Your contribution would be highly
> appreciated in this area.
>
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code. An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser. But of course, if we can find
> ways to make this task easier, we will.
>
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
>
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of
> metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
> parser definition to Monit.
>
> *Parsers* - The start script in
> `metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
> will give you good hooks into how each of the parsers are started.
>
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`. To understand
> that process, start at `tasks/main.yml`.
>
> I probably missed something, but let me know if you have questions.
>
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> �My wish, is that when I do an ansible-playbook -v -i {my configuration}
>> �metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
>> �my parser / topology is deployed, started and monitored the same way as
>
> the
>> �current bro, snort, and yaf parsers are.
>>
>> �I might be misunderstanding something however. I seems to me that all the
>> �examples of adding other parsers are temporary and not permanent because
>> �they do not have the full deployment, kind of push the config and run the
>> �script and you are going. Am I missing something? Would the squid sample
>> �steps result in a parser topology that would survive restarts / reboots
>> �etc?
>>
>> �On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
>> �wrote:
>>
>> �Just so I completely understand what you are asking for...you want to know
>> �how to create a new parser topology with the JSON parser and plug it into
>> �Monit so you can monitor and restart it on demand?
>>
>> �27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
>> �> Thanks James,
>> �>
>> �> I want to deploy an instance of the JSONMapParser into my POC cluster
>
> and
>> �vagrant. I\u2019m trying to work out exactly how to add a new configured parser
>> �instance to the deployment. I think these instructions would be a good
>> �extension to the squid stuff that is already there. If I could get that
>> �going and add a new parser all the way through, then maybe I can
>
> contribute
>> �something in that area. The ability to do this will also enable some of
>> �the other work you mentioned.
>> �>
>> �> On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
>> �wrote:
>> �>
>> �>> There are three types of parsers you can have currently. Our preferred
>> �way is to use Grok parser. The only thing you need to do there is to
>
> define
>> �your Grok statement and the parser will uptake it and do the rest. That is
>> �what most of our documentation reflect. The second type of parser that we
>> �have is a java parser, where you actually extend a parser class and define
>> �your own custom parsing logic. We intend this type of parser for high
>> �velocity feeds that require custom parsing logic that is not easily
>> �attainable by Grok. The third type of parser is the one you have been
>> �working on, a Json parser. This is a parser designed to take pre-parsed
>> �JSON for sensors that either log in JSON format natively or have been
>> �pre-parsed for us by some system upstream.
>> �>>
>> �>> Parsers don't integrate with Monit by default. We can come up with some
>> �instructions for you on how to do that.
>> �>>
>> �>> I should also note there are 2 additional parser types that are on the
>> �road map. METRON-295 (scripting bolt), which is a parser that allows you
>
> to
>> �uptake something like javascript, lua, etc., for doing the parsing. There
>> �is also METRON-288, which is a XSL parser designed to parse XML documents.
>> �If either of these are of interest to you we would welcome this
>> �contribution and we can work with you to get you started.
>> �>>
>> �>> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
>> �>>> Are all the steps required to add a parser documented anywhere? The
>> �squid
>> �>>> document starts the topology, but I don\u2019t think that integrates it in
>> �with
>> �>>> monit for example. Or does that actually happen?
>> �>>
>> �>> -------------------
>> �>> Thank you,
>> �>>
>> �>> James Sirota
>> �>> PPMC- Apache Metron (Incubating)
>> �>> jsirota AT apache DOT org
>>
>> �-------------------
>> �Thank you,
>>
>> �James Sirota
>> �PPMC- Apache Metron (Incubating)
>> �jsirota AT apache DOT org
>
> --
> Nick Allen <ni...@nickallen.org>
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Complete steps to add a new parser
Posted by Nick Allen <ni...@nickallen.org>.
Your changes to add the 'laa' sensor look "sensible." Nice.
On Sat, Oct 1, 2016 at 11:59 PM, Otto Fowler <ot...@gmail.com>
wrote:
> I have been able to add a new parser to the the deployment, and have the
> cluster fully deploy successfully. After I was able to push data to kafka
> from HDF and get it all indexed.
>
> Unlike quick dev and full, no problems getting the storm ports correct
> after deployment to my small cluster config.
>
> It looks to me that the steps I took to integrate the parser worked, but I
> still may have missed something. One thing that I know I missed was
> modifying the dashboard - adding the saved searches and integrating them
> with the visualizations.
>
> Here is a gist off a patch of my changes. The patch in the gist has been
> modified - so I don’t think it will apply for you. I removed proprietary
> field names ( the stellar config, the enrichment hosts, es index template ).
>
> I hope what is there is enough for you to verify, correct what I have done.
>
> https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84
>
>
>
> On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> Thanks Nick,
> That is some of the stuff that I have found trying to track down the
> deploy bits of the existing parsers, but I don’t want to miss anything, so
> I’d like some guidence. Right now, I’m OK with doing it all inside the
> metron ansible base. I expect that once I get it working and wrap my head
> around it I’ll have some ideas that I’ll float around improving this area,
> some use cases to propose that would possibly be external to the main
> deployment, or additive. First thing first is understanding all the points
> in doing it the hard way ;)
>
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 12:41:10, Nick Allen (nick@nickallen.org) wrote:
>
> Hi Otto -
>
> I would agree with you. We do not have documentation that describes how to
> 'permanently install' a new parser.. Your contribution would be highly
> appreciated in this area.
>
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code. An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser. But of course, if we can find
> ways to make this task easier, we will.
>
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
>
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of
> metron-deployment/roles/monit/templates/monit/parsers.monit to add your
> own
> parser definition to Monit.
>
> *Parsers* - The start script in
> `metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
> will give you good hooks into how each of the parsers are started.
>
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`. To understand
> that process, start at `tasks/main.yml`.
>
> I probably missed something, but let me know if you have questions.
>
>
>
>
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > My wish, is that when I do an ansible-playbook -v -i {my configuration}
> > metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> > my parser / topology is deployed, started and monitored the same way as
> the
> > current bro, snort, and yaf parsers are.
> >
> > I might be misunderstanding something however. I seems to me that all the
> > examples of adding other parsers are temporary and not permanent because
> > they do not have the full deployment, kind of push the config and run the
> > script and you are going. Am I missing something? Would the squid sample
> > steps result in a parser topology that would survive restarts / reboots
> > etc?
> >
> > On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
> > wrote:
> >
> > Just so I completely understand what you are asking for...you want to
> know
> > how to create a new parser topology with the JSON parser and plug it into
> > Monit so you can monitor and restart it on demand?
> >
> > 27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> > > Thanks James,
> > >
> > > I want to deploy an instance of the JSONMapParser into my POC cluster
> and
> > vagrant. I’m trying to work out exactly how to add a new configured
> parser
> > instance to the deployment. I think these instructions would be a good
> > extension to the squid stuff that is already there. If I could get that
> > going and add a new parser all the way through, then maybe I can
> contribute
> > something in that area. The ability to do this will also enable some of
> > the other work you mentioned.
> > >
> > > On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
> > wrote:
> > >
> > >> There are three types of parsers you can have currently. Our preferred
> > way is to use Grok parser. The only thing you need to do there is to
> define
> > your Grok statement and the parser will uptake it and do the rest. That
> is
> > what most of our documentation reflect. The second type of parser that we
> > have is a java parser, where you actually extend a parser class and
> define
> > your own custom parsing logic. We intend this type of parser for high
> > velocity feeds that require custom parsing logic that is not easily
> > attainable by Grok. The third type of parser is the one you have been
> > working on, a Json parser. This is a parser designed to take pre-parsed
> > JSON for sensors that either log in JSON format natively or have been
> > pre-parsed for us by some system upstream.
> > >>
> > >> Parsers don't integrate with Monit by default. We can come up with
> some
> > instructions for you on how to do that.
> > >>
> > >> I should also note there are 2 additional parser types that are on the
> > road map. METRON-295 (scripting bolt), which is a parser that allows you
> to
> > uptake something like javascript, lua, etc., for doing the parsing. There
> > is also METRON-288, which is a XSL parser designed to parse XML
> documents.
> > If either of these are of interest to you we would welcome this
> > contribution and we can work with you to get you started.
> > >>
> > >> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> > >>> Are all the steps required to add a parser documented anywhere? The
> > squid
> > >>> document starts the topology, but I don’t think that integrates it in
> > with
> > >>> monit for example. Or does that actually happen?
> > >>
> > >> -------------------
> > >> Thank you,
> > >>
> > >> James Sirota
> > >> PPMC- Apache Metron (Incubating)
> > >> jsirota AT apache DOT org
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
>
>
>
> --
> Nick Allen <ni...@nickallen.org>
>
>
--
Nick Allen <ni...@nickallen.org>
Re: Complete steps to add a new parser
Posted by Otto Fowler <ot...@gmail.com>.
I have been able to add a new parser to the the deployment, and have the
cluster fully deploy successfully. After I was able to push data to kafka
from HDF and get it all indexed.
Unlike quick dev and full, no problems getting the storm ports correct
after deployment to my small cluster config.
It looks to me that the steps I took to integrate the parser worked, but I
still may have missed something. One thing that I know I missed was
modifying the dashboard - adding the saved searches and integrating them
with the visualizations.
Here is a gist off a patch of my changes. The patch in the gist has been
modified - so I don’t think it will apply for you. I removed proprietary
field names ( the stellar config, the enrichment hosts, es index template ).
I hope what is there is enough for you to verify, correct what I have done.
https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84
On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwards@gmail.com)
wrote:
Thanks Nick,
That is some of the stuff that I have found trying to track down the deploy
bits of the existing parsers, but I don’t want to miss anything, so I’d
like some guidence. Right now, I’m OK with doing it all inside the metron
ansible base. I expect that once I get it working and wrap my head around
it I’ll have some ideas that I’ll float around improving this area, some
use cases to propose that would possibly be external to the main
deployment, or additive. First thing first is understanding all the points
in doing it the hard way ;)
--
Sent with Airmail
On September 27, 2016 at 12:41:10, Nick Allen (nick@nickallen.org) wrote:
Hi Otto -
I would agree with you. We do not have documentation that describes how to
'permanently install' a new parser.. Your contribution would be highly
appreciated in this area.
With the Ansible-based deployment of today, most likely you will have to
touch some of Metron's Ansible source code. An alternative would be to
mimic portions of Metron's deployment code, and manage that in its own
project, which would deploy your new parser. But of course, if we can find
ways to make this task easier, we will.
You may not have to touch each of these areas, but they at least will
provide you with a better understanding of how everything is stitched
together.
*Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
You can follow the pattern of
metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
parser definition to Monit.
*Parsers* - The start script in
`metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
will give you good hooks into how each of the parsers are started.
*Setup* - There are various setup tasks for the streaming functionality
that live under `metron-deployment/roles/metron_streaming`. To understand
that process, start at `tasks/main.yml`.
I probably missed something, but let me know if you have questions.
On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
wrote:
> My wish, is that when I do an ansible-playbook -v -i {my configuration}
> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> my parser / topology is deployed, started and monitored the same way as
the
> current bro, snort, and yaf parsers are.
>
> I might be misunderstanding something however. I seems to me that all the
> examples of adding other parsers are temporary and not permanent because
> they do not have the full deployment, kind of push the config and run the
> script and you are going. Am I missing something? Would the squid sample
> steps result in a parser topology that would survive restarts / reboots
> etc?
>
> On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
> wrote:
>
> Just so I completely understand what you are asking for...you want to know
> how to create a new parser topology with the JSON parser and plug it into
> Monit so you can monitor and restart it on demand?
>
> 27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> > Thanks James,
> >
> > I want to deploy an instance of the JSONMapParser into my POC cluster
and
> vagrant. I’m trying to work out exactly how to add a new configured parser
> instance to the deployment. I think these instructions would be a good
> extension to the squid stuff that is already there. If I could get that
> going and add a new parser all the way through, then maybe I can
contribute
> something in that area. The ability to do this will also enable some of
> the other work you mentioned.
> >
> > On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
> wrote:
> >
> >> There are three types of parsers you can have currently. Our preferred
> way is to use Grok parser. The only thing you need to do there is to
define
> your Grok statement and the parser will uptake it and do the rest. That is
> what most of our documentation reflect. The second type of parser that we
> have is a java parser, where you actually extend a parser class and define
> your own custom parsing logic. We intend this type of parser for high
> velocity feeds that require custom parsing logic that is not easily
> attainable by Grok. The third type of parser is the one you have been
> working on, a Json parser. This is a parser designed to take pre-parsed
> JSON for sensors that either log in JSON format natively or have been
> pre-parsed for us by some system upstream.
> >>
> >> Parsers don't integrate with Monit by default. We can come up with some
> instructions for you on how to do that.
> >>
> >> I should also note there are 2 additional parser types that are on the
> road map. METRON-295 (scripting bolt), which is a parser that allows you
to
> uptake something like javascript, lua, etc., for doing the parsing. There
> is also METRON-288, which is a XSL parser designed to parse XML documents.
> If either of these are of interest to you we would welcome this
> contribution and we can work with you to get you started.
> >>
> >> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> >>> Are all the steps required to add a parser documented anywhere? The
> squid
> >>> document starts the topology, but I don’t think that integrates it in
> with
> >>> monit for example. Or does that actually happen?
> >>
> >> -------------------
> >> Thank you,
> >>
> >> James Sirota
> >> PPMC- Apache Metron (Incubating)
> >> jsirota AT apache DOT org
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
--
Nick Allen <ni...@nickallen.org>
Re: Complete steps to add a new parser
Posted by Otto Fowler <ot...@gmail.com>.
Thanks Nick,
That is some of the stuff that I have found trying to track down the deploy
bits of the existing parsers, but I don’t want to miss anything, so I’d
like some guidence. Right now, I’m OK with doing it all inside the metron
ansible base. I expect that once I get it working and wrap my head around
it I’ll have some ideas that I’ll float around improving this area, some
use cases to propose that would possibly be external to the main
deployment, or additive. First thing first is understanding all the points
in doing it the hard way ;)
--
Sent with Airmail
On September 27, 2016 at 12:41:10, Nick Allen (nick@nickallen.org) wrote:
Hi Otto -
I would agree with you. We do not have documentation that describes how to
'permanently install' a new parser.. Your contribution would be highly
appreciated in this area.
With the Ansible-based deployment of today, most likely you will have to
touch some of Metron's Ansible source code. An alternative would be to
mimic portions of Metron's deployment code, and manage that in its own
project, which would deploy your new parser. But of course, if we can find
ways to make this task easier, we will.
You may not have to touch each of these areas, but they at least will
provide you with a better understanding of how everything is stitched
together.
*Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
You can follow the pattern of
metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
parser definition to Monit.
*Parsers* - The start script in
`metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
will give you good hooks into how each of the parsers are started.
*Setup* - There are various setup tasks for the streaming functionality
that live under `metron-deployment/roles/metron_streaming`. To understand
that process, start at `tasks/main.yml`.
I probably missed something, but let me know if you have questions.
On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
wrote:
> My wish, is that when I do an ansible-playbook -v -i {my configuration}
> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> my parser / topology is deployed, started and monitored the same way as
the
> current bro, snort, and yaf parsers are.
>
> I might be misunderstanding something however. I seems to me that all the
> examples of adding other parsers are temporary and not permanent because
> they do not have the full deployment, kind of push the config and run the
> script and you are going. Am I missing something? Would the squid sample
> steps result in a parser topology that would survive restarts / reboots
> etc?
>
> On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
> wrote:
>
> Just so I completely understand what you are asking for...you want to
know
> how to create a new parser topology with the JSON parser and plug it into
> Monit so you can monitor and restart it on demand?
>
> 27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> > Thanks James,
> >
> > I want to deploy an instance of the JSONMapParser into my POC cluster
and
> vagrant. I’m trying to work out exactly how to add a new configured
parser
> instance to the deployment. I think these instructions would be a good
> extension to the squid stuff that is already there. If I could get that
> going and add a new parser all the way through, then maybe I can
contribute
> something in that area. The ability to do this will also enable some of
> the other work you mentioned.
> >
> > On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
> wrote:
> >
> >> There are three types of parsers you can have currently. Our preferred
> way is to use Grok parser. The only thing you need to do there is to
define
> your Grok statement and the parser will uptake it and do the rest. That
is
> what most of our documentation reflect. The second type of parser that we
> have is a java parser, where you actually extend a parser class and
define
> your own custom parsing logic. We intend this type of parser for high
> velocity feeds that require custom parsing logic that is not easily
> attainable by Grok. The third type of parser is the one you have been
> working on, a Json parser. This is a parser designed to take pre-parsed
> JSON for sensors that either log in JSON format natively or have been
> pre-parsed for us by some system upstream.
> >>
> >> Parsers don't integrate with Monit by default. We can come up with
some
> instructions for you on how to do that.
> >>
> >> I should also note there are 2 additional parser types that are on the
> road map. METRON-295 (scripting bolt), which is a parser that allows you
to
> uptake something like javascript, lua, etc., for doing the parsing. There
> is also METRON-288, which is a XSL parser designed to parse XML
documents.
> If either of these are of interest to you we would welcome this
> contribution and we can work with you to get you started.
> >>
> >> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> >>> Are all the steps required to add a parser documented anywhere? The
> squid
> >>> document starts the topology, but I don’t think that integrates it in
> with
> >>> monit for example. Or does that actually happen?
> >>
> >> -------------------
> >> Thank you,
> >>
> >> James Sirota
> >> PPMC- Apache Metron (Incubating)
> >> jsirota AT apache DOT org
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
--
Nick Allen <ni...@nickallen.org>
Re: Complete steps to add a new parser
Posted by Nick Allen <ni...@nickallen.org>.
Hi Otto -
I would agree with you. We do not have documentation that describes how to
'permanently install' a new parser.. Your contribution would be highly
appreciated in this area.
With the Ansible-based deployment of today, most likely you will have to
touch some of Metron's Ansible source code. An alternative would be to
mimic portions of Metron's deployment code, and manage that in its own
project, which would deploy your new parser. But of course, if we can find
ways to make this task easier, we will.
You may not have to touch each of these areas, but they at least will
provide you with a better understanding of how everything is stitched
together.
*Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
You can follow the pattern of
metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
parser definition to Monit.
*Parsers* - The start script in
`metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
will give you good hooks into how each of the parsers are started.
*Setup* - There are various setup tasks for the streaming functionality
that live under `metron-deployment/roles/metron_streaming`. To understand
that process, start at `tasks/main.yml`.
I probably missed something, but let me know if you have questions.
On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ot...@gmail.com>
wrote:
> My wish, is that when I do an ansible-playbook -v -i {my configuration}
> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> my parser / topology is deployed, started and monitored the same way as the
> current bro, snort, and yaf parsers are.
>
> I might be misunderstanding something however. I seems to me that all the
> examples of adding other parsers are temporary and not permanent because
> they do not have the full deployment, kind of push the config and run the
> script and you are going. Am I missing something? Would the squid sample
> steps result in a parser topology that would survive restarts / reboots
> etc?
>
> On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org)
> wrote:
>
> Just so I completely understand what you are asking for...you want to know
> how to create a new parser topology with the JSON parser and plug it into
> Monit so you can monitor and restart it on demand?
>
> 27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> > Thanks James,
> >
> > I want to deploy an instance of the JSONMapParser into my POC cluster and
> vagrant. I’m trying to work out exactly how to add a new configured parser
> instance to the deployment. I think these instructions would be a good
> extension to the squid stuff that is already there. If I could get that
> going and add a new parser all the way through, then maybe I can contribute
> something in that area. The ability to do this will also enable some of
> the other work you mentioned.
> >
> > On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
> wrote:
> >
> >> There are three types of parsers you can have currently. Our preferred
> way is to use Grok parser. The only thing you need to do there is to define
> your Grok statement and the parser will uptake it and do the rest. That is
> what most of our documentation reflect. The second type of parser that we
> have is a java parser, where you actually extend a parser class and define
> your own custom parsing logic. We intend this type of parser for high
> velocity feeds that require custom parsing logic that is not easily
> attainable by Grok. The third type of parser is the one you have been
> working on, a Json parser. This is a parser designed to take pre-parsed
> JSON for sensors that either log in JSON format natively or have been
> pre-parsed for us by some system upstream.
> >>
> >> Parsers don't integrate with Monit by default. We can come up with some
> instructions for you on how to do that.
> >>
> >> I should also note there are 2 additional parser types that are on the
> road map. METRON-295 (scripting bolt), which is a parser that allows you to
> uptake something like javascript, lua, etc., for doing the parsing. There
> is also METRON-288, which is a XSL parser designed to parse XML documents.
> If either of these are of interest to you we would welcome this
> contribution and we can work with you to get you started.
> >>
> >> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> >>> Are all the steps required to add a parser documented anywhere? The
> squid
> >>> document starts the topology, but I don’t think that integrates it in
> with
> >>> monit for example. Or does that actually happen?
> >>
> >> -------------------
> >> Thank you,
> >>
> >> James Sirota
> >> PPMC- Apache Metron (Incubating)
> >> jsirota AT apache DOT org
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
--
Nick Allen <ni...@nickallen.org>
Re: Complete steps to add a new parser
Posted by Otto Fowler <ot...@gmail.com>.
My wish, is that when I do an ansible-playbook -v -i {my configuration}
metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
my parser / topology is deployed, started and monitored the same way as the
current bro, snort, and yaf parsers are.
I might be misunderstanding something however. I seems to me that all the
examples of adding other parsers are temporary and not permanent because
they do not have the full deployment, kind of push the config and run the
script and you are going. Am I missing something? Would the squid sample
steps result in a parser topology that would survive restarts / reboots etc?
On September 27, 2016 at 12:06:44, James Sirota (jsirota@apache.org) wrote:
Just so I completely understand what you are asking for...you want to know
how to create a new parser topology with the JSON parser and plug it into
Monit so you can monitor and restart it on demand?
27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> Thanks James,
>
> I want to deploy an instance of the JSONMapParser into my POC cluster and
vagrant. I’m trying to work out exactly how to add a new configured parser
instance to the deployment. I think these instructions would be a good
extension to the squid stuff that is already there. If I could get that
going and add a new parser all the way through, then maybe I can contribute
something in that area. The ability to do this will also enable some of
the other work you mentioned.
>
> On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org)
wrote:
>
>> There are three types of parsers you can have currently. Our preferred
way is to use Grok parser. The only thing you need to do there is to define
your Grok statement and the parser will uptake it and do the rest. That is
what most of our documentation reflect. The second type of parser that we
have is a java parser, where you actually extend a parser class and define
your own custom parsing logic. We intend this type of parser for high
velocity feeds that require custom parsing logic that is not easily
attainable by Grok. The third type of parser is the one you have been
working on, a Json parser. This is a parser designed to take pre-parsed
JSON for sensors that either log in JSON format natively or have been
pre-parsed for us by some system upstream.
>>
>> Parsers don't integrate with Monit by default. We can come up with some
instructions for you on how to do that.
>>
>> I should also note there are 2 additional parser types that are on the
road map. METRON-295 (scripting bolt), which is a parser that allows you to
uptake something like javascript, lua, etc., for doing the parsing. There
is also METRON-288, which is a XSL parser designed to parse XML documents.
If either of these are of interest to you we would welcome this
contribution and we can work with you to get you started.
>>
>> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
>>> Are all the steps required to add a parser documented anywhere? The
squid
>>> document starts the topology, but I don’t think that integrates it in
with
>>> monit for example. Or does that actually happen?
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Complete steps to add a new parser
Posted by James Sirota <js...@apache.org>.
Just so I completely understand what you are asking for...you want to know how to create a new parser topology with the JSON parser and plug it into Monit so you can monitor and restart it on demand?
27.09.2016, 09:03, "Otto Fowler" <ot...@gmail.com>:
> Thanks James,
>
> I want to deploy an instance of the JSONMapParser into my POC cluster and vagrant.� I\u2019m trying to work out exactly how to add a new configured parser instance to the deployment.� I think these instructions would be a good extension to the squid stuff that is already there.� If I could get that going and add a new parser all the way through, then maybe I can contribute something in that area.� The ability to do this will also enable some of the other work you mentioned.
>
> On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org) wrote:
>
>> There are three types of parsers you can have currently. Our preferred way is to use Grok parser. The only thing you need to do there is to define your Grok statement and the parser will uptake it and do the rest. That is what most of our documentation reflect. The second type of parser that we have is a java parser, where you actually extend a parser class and define your own custom parsing logic. We intend this type of parser for high velocity feeds that require custom parsing logic that is not easily attainable by Grok. The third type of parser is the one you have been working on, a Json parser. This is a parser designed to take pre-parsed JSON for sensors that either log in JSON format natively or have been pre-parsed for us by some system upstream.
>>
>> Parsers don't integrate with Monit by default. We can come up with some instructions for you on how to do that.
>>
>> I should also note there are 2 additional parser types that are on the road map. METRON-295 (scripting bolt), which is a parser that allows you to uptake something like javascript, lua, etc., for doing the parsing. There is also METRON-288, which is a XSL parser designed to parse XML documents. If either of these are of interest to you we would welcome this contribution and we can work with you to get you started.
>>
>> 26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
>>> Are all the steps required to add a parser documented anywhere? The squid
>>> document starts the topology, but I don\u2019t think that integrates it in with
>>> monit for example. Or does that actually happen?
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Complete steps to add a new parser
Posted by Otto Fowler <ot...@gmail.com>.
Thanks James,
I want to deploy an instance of the JSONMapParser into my POC cluster and
vagrant. I’m trying to work out exactly how to add a new configured parser
instance to the deployment. I think these instructions would be a good
extension to the squid stuff that is already there. If I could get that
going and add a new parser all the way through, then maybe I can contribute
something in that area. The ability to do this will also enable some of
the other work you mentioned.
On September 27, 2016 at 11:51:41, James Sirota (jsirota@apache.org) wrote:
There are three types of parsers you can have currently. Our preferred way
is to use Grok parser. The only thing you need to do there is to define
your Grok statement and the parser will uptake it and do the rest. That is
what most of our documentation reflect. The second type of parser that we
have is a java parser, where you actually extend a parser class and define
your own custom parsing logic. We intend this type of parser for high
velocity feeds that require custom parsing logic that is not easily
attainable by Grok. The third type of parser is the one you have been
working on, a Json parser. This is a parser designed to take pre-parsed
JSON for sensors that either log in JSON format natively or have been
pre-parsed for us by some system upstream.
Parsers don't integrate with Monit by default. We can come up with some
instructions for you on how to do that.
I should also note there are 2 additional parser types that are on the road
map. METRON-295 (scripting bolt), which is a parser that allows you to
uptake something like javascript, lua, etc., for doing the parsing. There
is also METRON-288, which is a XSL parser designed to parse XML documents.
If either of these are of interest to you we would welcome this
contribution and we can work with you to get you started.
26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> Are all the steps required to add a parser documented anywhere? The squid
> document starts the topology, but I don’t think that integrates it in
with
> monit for example. Or does that actually happen?
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: Complete steps to add a new parser
Posted by James Sirota <js...@apache.org>.
There are three types of parsers you can have currently. Our preferred way is to use Grok parser. The only thing you need to do there is to define your Grok statement and the parser will uptake it and do the rest. That is what most of our documentation reflect. The second type of parser that we have is a java parser, where you actually extend a parser class and define your own custom parsing logic. We intend this type of parser for high velocity feeds that require custom parsing logic that is not easily attainable by Grok. The third type of parser is the one you have been working on, a Json parser. This is a parser designed to take pre-parsed JSON for sensors that either log in JSON format natively or have been pre-parsed for us by some system upstream.
Parsers don't integrate with Monit by default. We can come up with some instructions for you on how to do that.
I should also note there are 2 additional parser types that are on the road map. METRON-295 (scripting bolt), which is a parser that allows you to uptake something like javascript, lua, etc., for doing the parsing. There is also METRON-288, which is a XSL parser designed to parse XML documents. If either of these are of interest to you we would welcome this contribution and we can work with you to get you started.
26.09.2016, 10:35, "Otto Fowler" <ot...@gmail.com>:
> Are all the steps required to add a parser documented anywhere? The squid
> document starts the topology, but I don\u2019t think that integrates it in with
> monit for example. Or does that actually happen?
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org