You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2017/09/20 13:39:38 UTC
[DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
The question has come up about the metron parsers installation vs. parser
extension installation differences, and I’d like to get some comments.
Right now ( let’s pretend the UI PR get’s merged to the feature branch for
a minute ) in the original take on this the metron parsers ( bro, yaf,
snort etc ) get installed
into the system basically as before with regards to zookeeper ( although
they ALL get installed since they all have demo compatible default configs
now). The parser extensions, installed after the fact through ui however
have a new parser extension configuration
that is registered into a new zookeeper area, and are listed and managed in
the new UI. They can be installed or uninstalled basically.
The question is if the parsers installed by metron should *also* be
registered in the same way, such that the parser extension ui lists all of
the parsers.
This would allow removal and installation of metron parsers installed by
the system. Also, following on we can customize the install to not install
everything. It may also be more simple concept wise.
That is the basic thing. So the question is if we want to go in this
direction.
The not so basic thing is to still deploy the extensions into the system as
packages, but not install them, such that you can add an extension from a
file *and also* from a ‘repository’ of
extensions. Down the line we can support local and remote repositories etc.
So the options are:
1. As it is now on the feature branch ( still pretending ;)), the system
installed parsers are not the same as the extension installed parsers.
While the project can uninstall and replace these parsers, the user/ui
cannot.
2. All parsers are installed as extensions and can be removed, but are all
initially installed
3. All parsers are extensions, but not installed during the original
deployment, rather they are put into a repository that the ui lets you
browse to select, in addition to allowing
install from file ( like intellij and other plugin systems do ). **This
would have implications for the demo system, since we would want to still
install the bro, snort and maybe yaf parsers.
In considering these questions, we need to keep in mind where we want to
go, and how much of this is required for first release of the extension
system. There is a lot of ‘while we are already doing xxx, we might as
well do yyyy since it will be harder later’ in this.
Thanks for your time and your timely responses.
ottO
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Per our prior conversations, I prefer option 2 - treating third party and
built-in the same way. I would love to see signing of extensions in the
future as a potential follow-on so we could verify the Metron built-ins
(and even third parties).
Jon
On Wed, Sep 20, 2017 at 10:22 AM Otto Fowler <ot...@gmail.com>
wrote:
> Simon, I’m sorry, I may not have answered your question.
> I use parser and sensors as the same thing, but from what you say I think I
> mean parser.
>
>
> On September 20, 2017 at 10:08:25, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> So,
> The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if
> it is defined somewhere and I have missed it, please point me in the right
> direction.
> While I don’t want to fork the discussion, the question is where you find
> it so to speak, so about bundles and configurations.
>
> With the extension system you have two options for ‘config’ only
> parser/sensor installs.
>
> 1. The previous mechanism of creating the configurations and manually
> pushing them zookeeper and possibly patterns still works, although they
> will not be managed as extensions. This I believe is a feature gap that
> already exists and I did not address it as part of this effort.
> 2. Create a new parser from the archetype, but remove all the src/main code
> and make it configuration only. This I think should be the recommended way
> to create configuration only parsers, since such parsers will still have
> the unit and integration tests. Flows where you start with the archetype
> and refine through tests or manually deploy, refine and then build from the
> archetype can be imagined.
>
>
> The main question for this thread however, is should metron parsers and
> 3rd party parsers be treated the same -> they are all extensions,
> manageable through the extensions ui.
>
> I can demo for you whenever you are free if you don’t want to wait for the
> community demo.
>
>
> On September 20, 2017 at 09:52:56, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> Otto,
>
> Can you just clarify what you mean by parsers in this instance. To my mind
> parsers in metron are be classes, and do not have any configuration
> settings. Instances of parsers are referred to in the ui as sensors, and
> are essentially concrete instances of parsers and as such do have config.
>
> The parser vs sensor distinction feels like a valuable one to make here.
> I'd really like a clearer understanding of the role of bundles in config,
> and how we maintain the ability to control config without the need for
> files in the bundle.
>
> Maybe some samples / a demo would really clear this up, but to be honest
> right now I'm a little confused about how this would work for an average
> SOC team who do not have maven available.
>
> Thanks,
> Simon
>
> Sent from my iPhone
>
> > On 20 Sep 2017, at 23:43, Otto Fowler <ot...@gmail.com> wrote:
> >
> > Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> > part of the system and not installed individually.
> >
> >
> >
> > On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> > The question has come up about the metron parsers installation vs. parser
> > extension installation differences, and I’d like to get some comments.
> >
> > Right now ( let’s pretend the UI PR get’s merged to the feature branch
> for
> > a minute ) in the original take on this the metron parsers ( bro, yaf,
> > snort etc ) get installed
> > into the system basically as before with regards to zookeeper ( although
> > they ALL get installed since they all have demo compatible default
> configs
> > now). The parser extensions, installed after the fact through ui however
> > have a new parser extension configuration
> > that is registered into a new zookeeper area, and are listed and managed
> in
> > the new UI. They can be installed or uninstalled basically.
> >
> > The question is if the parsers installed by metron should *also* be
> > registered in the same way, such that the parser extension ui lists all
> of
> > the parsers.
> >
> > This would allow removal and installation of metron parsers installed by
> > the system. Also, following on we can customize the install to not
> install
> > everything. It may also be more simple concept wise.
> >
> > That is the basic thing. So the question is if we want to go in this
> > direction.
> >
> > The not so basic thing is to still deploy the extensions into the system
> as
> > packages, but not install them, such that you can add an extension from a
> > file *and also* from a ‘repository’ of
> > extensions. Down the line we can support local and remote repositories
> etc.
> >
> > So the options are:
> >
> > 1. As it is now on the feature branch ( still pretending ;)), the system
> > installed parsers are not the same as the extension installed parsers.
> > While the project can uninstall and replace these parsers, the user/ui
> > cannot.
> > 2. All parsers are installed as extensions and can be removed, but are
> all
> > initially installed
> > 3. All parsers are extensions, but not installed during the original
> > deployment, rather they are put into a repository that the ui lets you
> > browse to select, in addition to allowing
> > install from file ( like intellij and other plugin systems do ). **This
> > would have implications for the demo system, since we would want to still
> > install the bro, snort and maybe yaf parsers.
> >
> >
> > In considering these questions, we need to keep in mind where we want to
> > go, and how much of this is required for first release of the extension
> > system. There is a lot of ‘while we are already doing xxx, we might as
> > well do yyyy since it will be harder later’ in this.
> >
> > Thanks for your time and your timely responses.
> >
> > ottO
>
--
Jon
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by Otto Fowler <ot...@gmail.com>.
OK,
So, I think that this discussion should be taken up again after the demo.
It will be hopefully
easier then.
Sorry for the static.
Also : remember
https://cwiki.apache.org/confluence/display/METRON/Metron+Extension+System+and+Parser+Extensions
On September 20, 2017 at 14:29:51, Ryan Merriman (merrimanr@gmail.com)
wrote:
I will attempt to clarify parsers vs sensors. Parsers refer to concrete
parser classes and sensors refer to configuration + one of the parser
classes (with parser class being defined in the configuration). The
architecture was designed so that a parser class can be made dynamic and
behave differently depending on configuration. This allows multiple use
cases to be handled by a single parser class with different
configurations. A good example of this (and the primary reason this
pattern was chosen) is the GrokParser. This parser can handle multiple
sensors even though it is the same parser class in each instance.
What are some reasons we would want to use the bundle system to deploy a
configuration only sensor? From what I understand creating a bundle is not
a simple process and requires manually editing configuration files (correct
me if I am wrong). The whole reason we created the management UI was to
allow people to create sensors without having to go through this difficult,
error-prone process that initially made Metron hard to use. You can create
sensors right now with the management UI as long as the necessary parser
class is available. You mention test coverage but I would argue the unit
and integration tests for that parser class should cover the edge cases of
configuration. What happens if I change the configuration? Do I have to
also go update the unit/integration tests for that parser? Do I have to
rebuild and reinstall/update the parser?
This bundle system is absolutely necessary for parser classes but for me it
feels a little too tightly coupled to configuration. Not saying
configuration shouldn't be involved at all, there is definitely value in
providing a default/bootstrap configuration for a parser class for someone
to start with. Creating a parser class or a bundle is a complex task
suited for an engineer (with the help of the community). A SOC analyst
should only have to understand the various configuration options to create
and maintain a sensor and be able to make these changes through a user
interface.
This feature branch moves pretty fast so it's entirely possible my
knowledge is incorrect or outdated. Don't hesitate to correct anything I'm
missing or have incorrect. My main concern is that adding this feature
should make Metron easier to use and not harder. Is it possible these 2
options could coexist?
Ryan
On Wed, Sep 20, 2017 at 9:21 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Simon, I’m sorry, I may not have answered your question.
> I use parser and sensors as the same thing, but from what you say I think
I
> mean parser.
>
>
> On September 20, 2017 at 10:08:25, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> So,
> The distinction between ‘sensor’ and ‘parser’ is not clear to me either,
if
> it is defined somewhere and I have missed it, please point me in the
right
> direction.
> While I don’t want to fork the discussion, the question is where you find
> it so to speak, so about bundles and configurations.
>
> With the extension system you have two options for ‘config’ only
> parser/sensor installs.
>
> 1. The previous mechanism of creating the configurations and manually
> pushing them zookeeper and possibly patterns still works, although they
> will not be managed as extensions. This I believe is a feature gap that
> already exists and I did not address it as part of this effort.
> 2. Create a new parser from the archetype, but remove all the src/main
code
> and make it configuration only. This I think should be the recommended
way
> to create configuration only parsers, since such parsers will still have
> the unit and integration tests. Flows where you start with the archetype
> and refine through tests or manually deploy, refine and then build from
the
> archetype can be imagined.
>
>
> The main question for this thread however, is should metron parsers and
> 3rd party parsers be treated the same -> they are all extensions,
> manageable through the extensions ui.
>
> I can demo for you whenever you are free if you don’t want to wait for
the
> community demo.
>
>
> On September 20, 2017 at 09:52:56, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> Otto,
>
> Can you just clarify what you mean by parsers in this instance. To my
mind
> parsers in metron are be classes, and do not have any configuration
> settings. Instances of parsers are referred to in the ui as sensors, and
> are essentially concrete instances of parsers and as such do have config.
>
> The parser vs sensor distinction feels like a valuable one to make here.
> I'd really like a clearer understanding of the role of bundles in config,
> and how we maintain the ability to control config without the need for
> files in the bundle.
>
> Maybe some samples / a demo would really clear this up, but to be honest
> right now I'm a little confused about how this would work for an average
> SOC team who do not have maven available.
>
> Thanks,
> Simon
>
> Sent from my iPhone
>
> > On 20 Sep 2017, at 23:43, Otto Fowler <ot...@gmail.com> wrote:
> >
> > Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> > part of the system and not installed individually.
> >
> >
> >
> > On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> > The question has come up about the metron parsers installation vs.
parser
> > extension installation differences, and I’d like to get some comments.
> >
> > Right now ( let’s pretend the UI PR get’s merged to the feature branch
> for
> > a minute ) in the original take on this the metron parsers ( bro, yaf,
> > snort etc ) get installed
> > into the system basically as before with regards to zookeeper (
although
> > they ALL get installed since they all have demo compatible default
> configs
> > now). The parser extensions, installed after the fact through ui
however
> > have a new parser extension configuration
> > that is registered into a new zookeeper area, and are listed and
managed
> in
> > the new UI. They can be installed or uninstalled basically.
> >
> > The question is if the parsers installed by metron should *also* be
> > registered in the same way, such that the parser extension ui lists all
> of
> > the parsers.
> >
> > This would allow removal and installation of metron parsers installed
by
> > the system. Also, following on we can customize the install to not
> install
> > everything. It may also be more simple concept wise.
> >
> > That is the basic thing. So the question is if we want to go in this
> > direction.
> >
> > The not so basic thing is to still deploy the extensions into the
system
> as
> > packages, but not install them, such that you can add an extension from
a
> > file *and also* from a ‘repository’ of
> > extensions. Down the line we can support local and remote repositories
> etc.
> >
> > So the options are:
> >
> > 1. As it is now on the feature branch ( still pretending ;)), the
system
> > installed parsers are not the same as the extension installed parsers.
> > While the project can uninstall and replace these parsers, the user/ui
> > cannot.
> > 2. All parsers are installed as extensions and can be removed, but are
> all
> > initially installed
> > 3. All parsers are extensions, but not installed during the original
> > deployment, rather they are put into a repository that the ui lets you
> > browse to select, in addition to allowing
> > install from file ( like intellij and other plugin systems do ). **This
> > would have implications for the demo system, since we would want to
still
> > install the bro, snort and maybe yaf parsers.
> >
> >
> > In considering these questions, we need to keep in mind where we want
to
> > go, and how much of this is required for first release of the extension
> > system. There is a lot of ‘while we are already doing xxx, we might as
> > well do yyyy since it will be harder later’ in this.
> >
> > Thanks for your time and your timely responses.
> >
> > ottO
>
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by Ryan Merriman <me...@gmail.com>.
I will attempt to clarify parsers vs sensors. Parsers refer to concrete
parser classes and sensors refer to configuration + one of the parser
classes (with parser class being defined in the configuration). The
architecture was designed so that a parser class can be made dynamic and
behave differently depending on configuration. This allows multiple use
cases to be handled by a single parser class with different
configurations. A good example of this (and the primary reason this
pattern was chosen) is the GrokParser. This parser can handle multiple
sensors even though it is the same parser class in each instance.
What are some reasons we would want to use the bundle system to deploy a
configuration only sensor? From what I understand creating a bundle is not
a simple process and requires manually editing configuration files (correct
me if I am wrong). The whole reason we created the management UI was to
allow people to create sensors without having to go through this difficult,
error-prone process that initially made Metron hard to use. You can create
sensors right now with the management UI as long as the necessary parser
class is available. You mention test coverage but I would argue the unit
and integration tests for that parser class should cover the edge cases of
configuration. What happens if I change the configuration? Do I have to
also go update the unit/integration tests for that parser? Do I have to
rebuild and reinstall/update the parser?
This bundle system is absolutely necessary for parser classes but for me it
feels a little too tightly coupled to configuration. Not saying
configuration shouldn't be involved at all, there is definitely value in
providing a default/bootstrap configuration for a parser class for someone
to start with. Creating a parser class or a bundle is a complex task
suited for an engineer (with the help of the community). A SOC analyst
should only have to understand the various configuration options to create
and maintain a sensor and be able to make these changes through a user
interface.
This feature branch moves pretty fast so it's entirely possible my
knowledge is incorrect or outdated. Don't hesitate to correct anything I'm
missing or have incorrect. My main concern is that adding this feature
should make Metron easier to use and not harder. Is it possible these 2
options could coexist?
Ryan
On Wed, Sep 20, 2017 at 9:21 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Simon, I’m sorry, I may not have answered your question.
> I use parser and sensors as the same thing, but from what you say I think I
> mean parser.
>
>
> On September 20, 2017 at 10:08:25, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> So,
> The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if
> it is defined somewhere and I have missed it, please point me in the right
> direction.
> While I don’t want to fork the discussion, the question is where you find
> it so to speak, so about bundles and configurations.
>
> With the extension system you have two options for ‘config’ only
> parser/sensor installs.
>
> 1. The previous mechanism of creating the configurations and manually
> pushing them zookeeper and possibly patterns still works, although they
> will not be managed as extensions. This I believe is a feature gap that
> already exists and I did not address it as part of this effort.
> 2. Create a new parser from the archetype, but remove all the src/main code
> and make it configuration only. This I think should be the recommended way
> to create configuration only parsers, since such parsers will still have
> the unit and integration tests. Flows where you start with the archetype
> and refine through tests or manually deploy, refine and then build from the
> archetype can be imagined.
>
>
> The main question for this thread however, is should metron parsers and
> 3rd party parsers be treated the same -> they are all extensions,
> manageable through the extensions ui.
>
> I can demo for you whenever you are free if you don’t want to wait for the
> community demo.
>
>
> On September 20, 2017 at 09:52:56, Simon Elliston Ball (
> simon@simonellistonball.com) wrote:
>
> Otto,
>
> Can you just clarify what you mean by parsers in this instance. To my mind
> parsers in metron are be classes, and do not have any configuration
> settings. Instances of parsers are referred to in the ui as sensors, and
> are essentially concrete instances of parsers and as such do have config.
>
> The parser vs sensor distinction feels like a valuable one to make here.
> I'd really like a clearer understanding of the role of bundles in config,
> and how we maintain the ability to control config without the need for
> files in the bundle.
>
> Maybe some samples / a demo would really clear this up, but to be honest
> right now I'm a little confused about how this would work for an average
> SOC team who do not have maven available.
>
> Thanks,
> Simon
>
> Sent from my iPhone
>
> > On 20 Sep 2017, at 23:43, Otto Fowler <ot...@gmail.com> wrote:
> >
> > Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> > part of the system and not installed individually.
> >
> >
> >
> > On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> > The question has come up about the metron parsers installation vs. parser
> > extension installation differences, and I’d like to get some comments.
> >
> > Right now ( let’s pretend the UI PR get’s merged to the feature branch
> for
> > a minute ) in the original take on this the metron parsers ( bro, yaf,
> > snort etc ) get installed
> > into the system basically as before with regards to zookeeper ( although
> > they ALL get installed since they all have demo compatible default
> configs
> > now). The parser extensions, installed after the fact through ui however
> > have a new parser extension configuration
> > that is registered into a new zookeeper area, and are listed and managed
> in
> > the new UI. They can be installed or uninstalled basically.
> >
> > The question is if the parsers installed by metron should *also* be
> > registered in the same way, such that the parser extension ui lists all
> of
> > the parsers.
> >
> > This would allow removal and installation of metron parsers installed by
> > the system. Also, following on we can customize the install to not
> install
> > everything. It may also be more simple concept wise.
> >
> > That is the basic thing. So the question is if we want to go in this
> > direction.
> >
> > The not so basic thing is to still deploy the extensions into the system
> as
> > packages, but not install them, such that you can add an extension from a
> > file *and also* from a ‘repository’ of
> > extensions. Down the line we can support local and remote repositories
> etc.
> >
> > So the options are:
> >
> > 1. As it is now on the feature branch ( still pretending ;)), the system
> > installed parsers are not the same as the extension installed parsers.
> > While the project can uninstall and replace these parsers, the user/ui
> > cannot.
> > 2. All parsers are installed as extensions and can be removed, but are
> all
> > initially installed
> > 3. All parsers are extensions, but not installed during the original
> > deployment, rather they are put into a repository that the ui lets you
> > browse to select, in addition to allowing
> > install from file ( like intellij and other plugin systems do ). **This
> > would have implications for the demo system, since we would want to still
> > install the bro, snort and maybe yaf parsers.
> >
> >
> > In considering these questions, we need to keep in mind where we want to
> > go, and how much of this is required for first release of the extension
> > system. There is a lot of ‘while we are already doing xxx, we might as
> > well do yyyy since it will be harder later’ in this.
> >
> > Thanks for your time and your timely responses.
> >
> > ottO
>
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by Otto Fowler <ot...@gmail.com>.
Simon, I’m sorry, I may not have answered your question.
I use parser and sensors as the same thing, but from what you say I think I
mean parser.
On September 20, 2017 at 10:08:25, Otto Fowler (ottobackwards@gmail.com)
wrote:
So,
The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if
it is defined somewhere and I have missed it, please point me in the right
direction.
While I don’t want to fork the discussion, the question is where you find
it so to speak, so about bundles and configurations.
With the extension system you have two options for ‘config’ only
parser/sensor installs.
1. The previous mechanism of creating the configurations and manually
pushing them zookeeper and possibly patterns still works, although they
will not be managed as extensions. This I believe is a feature gap that
already exists and I did not address it as part of this effort.
2. Create a new parser from the archetype, but remove all the src/main code
and make it configuration only. This I think should be the recommended way
to create configuration only parsers, since such parsers will still have
the unit and integration tests. Flows where you start with the archetype
and refine through tests or manually deploy, refine and then build from the
archetype can be imagined.
The main question for this thread however, is should metron parsers and
3rd party parsers be treated the same -> they are all extensions,
manageable through the extensions ui.
I can demo for you whenever you are free if you don’t want to wait for the
community demo.
On September 20, 2017 at 09:52:56, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Otto,
Can you just clarify what you mean by parsers in this instance. To my mind
parsers in metron are be classes, and do not have any configuration
settings. Instances of parsers are referred to in the ui as sensors, and
are essentially concrete instances of parsers and as such do have config.
The parser vs sensor distinction feels like a valuable one to make here.
I'd really like a clearer understanding of the role of bundles in config,
and how we maintain the ability to control config without the need for
files in the bundle.
Maybe some samples / a demo would really clear this up, but to be honest
right now I'm a little confused about how this would work for an average
SOC team who do not have maven available.
Thanks,
Simon
Sent from my iPhone
> On 20 Sep 2017, at 23:43, Otto Fowler <ot...@gmail.com> wrote:
>
> Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> part of the system and not installed individually.
>
>
>
> On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> The question has come up about the metron parsers installation vs. parser
> extension installation differences, and I’d like to get some comments.
>
> Right now ( let’s pretend the UI PR get’s merged to the feature branch for
> a minute ) in the original take on this the metron parsers ( bro, yaf,
> snort etc ) get installed
> into the system basically as before with regards to zookeeper ( although
> they ALL get installed since they all have demo compatible default configs
> now). The parser extensions, installed after the fact through ui however
> have a new parser extension configuration
> that is registered into a new zookeeper area, and are listed and managed
in
> the new UI. They can be installed or uninstalled basically.
>
> The question is if the parsers installed by metron should *also* be
> registered in the same way, such that the parser extension ui lists all of
> the parsers.
>
> This would allow removal and installation of metron parsers installed by
> the system. Also, following on we can customize the install to not install
> everything. It may also be more simple concept wise.
>
> That is the basic thing. So the question is if we want to go in this
> direction.
>
> The not so basic thing is to still deploy the extensions into the system
as
> packages, but not install them, such that you can add an extension from a
> file *and also* from a ‘repository’ of
> extensions. Down the line we can support local and remote repositories
etc.
>
> So the options are:
>
> 1. As it is now on the feature branch ( still pretending ;)), the system
> installed parsers are not the same as the extension installed parsers.
> While the project can uninstall and replace these parsers, the user/ui
> cannot.
> 2. All parsers are installed as extensions and can be removed, but are all
> initially installed
> 3. All parsers are extensions, but not installed during the original
> deployment, rather they are put into a repository that the ui lets you
> browse to select, in addition to allowing
> install from file ( like intellij and other plugin systems do ). **This
> would have implications for the demo system, since we would want to still
> install the bro, snort and maybe yaf parsers.
>
>
> In considering these questions, we need to keep in mind where we want to
> go, and how much of this is required for first release of the extension
> system. There is a lot of ‘while we are already doing xxx, we might as
> well do yyyy since it will be harder later’ in this.
>
> Thanks for your time and your timely responses.
>
> ottO
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by Otto Fowler <ot...@gmail.com>.
So,
The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if
it is defined somewhere and I have missed it, please point me in the right
direction.
While I don’t want to fork the discussion, the question is where you find
it so to speak, so about bundles and configurations.
With the extension system you have two options for ‘config’ only
parser/sensor installs.
1. The previous mechanism of creating the configurations and manually
pushing them zookeeper and possibly patterns still works, although they
will not be managed as extensions. This I believe is a feature gap that
already exists and I did not address it as part of this effort.
2. Create a new parser from the archetype, but remove all the src/main code
and make it configuration only. This I think should be the recommended way
to create configuration only parsers, since such parsers will still have
the unit and integration tests. Flows where you start with the archetype
and refine through tests or manually deploy, refine and then build from the
archetype can be imagined.
The main question for this thread however, is should metron parsers and
3rd party parsers be treated the same -> they are all extensions,
manageable through the extensions ui.
I can demo for you whenever you are free if you don’t want to wait for the
community demo.
On September 20, 2017 at 09:52:56, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Otto,
Can you just clarify what you mean by parsers in this instance. To my mind
parsers in metron are be classes, and do not have any configuration
settings. Instances of parsers are referred to in the ui as sensors, and
are essentially concrete instances of parsers and as such do have config.
The parser vs sensor distinction feels like a valuable one to make here.
I'd really like a clearer understanding of the role of bundles in config,
and how we maintain the ability to control config without the need for
files in the bundle.
Maybe some samples / a demo would really clear this up, but to be honest
right now I'm a little confused about how this would work for an average
SOC team who do not have maven available.
Thanks,
Simon
Sent from my iPhone
> On 20 Sep 2017, at 23:43, Otto Fowler <ot...@gmail.com> wrote:
>
> Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> part of the system and not installed individually.
>
>
>
> On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> The question has come up about the metron parsers installation vs. parser
> extension installation differences, and I’d like to get some comments.
>
> Right now ( let’s pretend the UI PR get’s merged to the feature branch
for
> a minute ) in the original take on this the metron parsers ( bro, yaf,
> snort etc ) get installed
> into the system basically as before with regards to zookeeper ( although
> they ALL get installed since they all have demo compatible default
configs
> now). The parser extensions, installed after the fact through ui however
> have a new parser extension configuration
> that is registered into a new zookeeper area, and are listed and managed
in
> the new UI. They can be installed or uninstalled basically.
>
> The question is if the parsers installed by metron should *also* be
> registered in the same way, such that the parser extension ui lists all
of
> the parsers.
>
> This would allow removal and installation of metron parsers installed by
> the system. Also, following on we can customize the install to not
install
> everything. It may also be more simple concept wise.
>
> That is the basic thing. So the question is if we want to go in this
> direction.
>
> The not so basic thing is to still deploy the extensions into the system
as
> packages, but not install them, such that you can add an extension from a
> file *and also* from a ‘repository’ of
> extensions. Down the line we can support local and remote repositories
etc.
>
> So the options are:
>
> 1. As it is now on the feature branch ( still pretending ;)), the system
> installed parsers are not the same as the extension installed parsers.
> While the project can uninstall and replace these parsers, the user/ui
> cannot.
> 2. All parsers are installed as extensions and can be removed, but are
all
> initially installed
> 3. All parsers are extensions, but not installed during the original
> deployment, rather they are put into a repository that the ui lets you
> browse to select, in addition to allowing
> install from file ( like intellij and other plugin systems do ). **This
> would have implications for the demo system, since we would want to still
> install the bro, snort and maybe yaf parsers.
>
>
> In considering these questions, we need to keep in mind where we want to
> go, and how much of this is required for first release of the extension
> system. There is a lot of ‘while we are already doing xxx, we might as
> well do yyyy since it will be harder later’ in this.
>
> Thanks for your time and your timely responses.
>
> ottO
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Otto,
Can you just clarify what you mean by parsers in this instance. To my mind parsers in metron are be classes, and do not have any configuration settings. Instances of parsers are referred to in the ui as sensors, and are essentially concrete instances of parsers and as such do have config.
The parser vs sensor distinction feels like a valuable one to make here. I'd really like a clearer understanding of the role of bundles in config, and how we maintain the ability to control config without the need for files in the bundle.
Maybe some samples / a demo would really clear this up, but to be honest right now I'm a little confused about how this would work for an average SOC team who do not have maven available.
Thanks,
Simon
Sent from my iPhone
> On 20 Sep 2017, at 23:43, Otto Fowler <ot...@gmail.com> wrote:
>
> Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> part of the system and not installed individually.
>
>
>
> On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> The question has come up about the metron parsers installation vs. parser
> extension installation differences, and I’d like to get some comments.
>
> Right now ( let’s pretend the UI PR get’s merged to the feature branch for
> a minute ) in the original take on this the metron parsers ( bro, yaf,
> snort etc ) get installed
> into the system basically as before with regards to zookeeper ( although
> they ALL get installed since they all have demo compatible default configs
> now). The parser extensions, installed after the fact through ui however
> have a new parser extension configuration
> that is registered into a new zookeeper area, and are listed and managed in
> the new UI. They can be installed or uninstalled basically.
>
> The question is if the parsers installed by metron should *also* be
> registered in the same way, such that the parser extension ui lists all of
> the parsers.
>
> This would allow removal and installation of metron parsers installed by
> the system. Also, following on we can customize the install to not install
> everything. It may also be more simple concept wise.
>
> That is the basic thing. So the question is if we want to go in this
> direction.
>
> The not so basic thing is to still deploy the extensions into the system as
> packages, but not install them, such that you can add an extension from a
> file *and also* from a ‘repository’ of
> extensions. Down the line we can support local and remote repositories etc.
>
> So the options are:
>
> 1. As it is now on the feature branch ( still pretending ;)), the system
> installed parsers are not the same as the extension installed parsers.
> While the project can uninstall and replace these parsers, the user/ui
> cannot.
> 2. All parsers are installed as extensions and can be removed, but are all
> initially installed
> 3. All parsers are extensions, but not installed during the original
> deployment, rather they are put into a repository that the ui lets you
> browse to select, in addition to allowing
> install from file ( like intellij and other plugin systems do ). **This
> would have implications for the demo system, since we would want to still
> install the bro, snort and maybe yaf parsers.
>
>
> In considering these questions, we need to keep in mind where we want to
> go, and how much of this is required for first release of the extension
> system. There is a lot of ‘while we are already doing xxx, we might as
> well do yyyy since it will be harder later’ in this.
>
> Thanks for your time and your timely responses.
>
> ottO
Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions
Posted by Otto Fowler <ot...@gmail.com>.
Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
part of the system and not installed individually.
On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwards@gmail.com)
wrote:
The question has come up about the metron parsers installation vs. parser
extension installation differences, and I’d like to get some comments.
Right now ( let’s pretend the UI PR get’s merged to the feature branch for
a minute ) in the original take on this the metron parsers ( bro, yaf,
snort etc ) get installed
into the system basically as before with regards to zookeeper ( although
they ALL get installed since they all have demo compatible default configs
now). The parser extensions, installed after the fact through ui however
have a new parser extension configuration
that is registered into a new zookeeper area, and are listed and managed in
the new UI. They can be installed or uninstalled basically.
The question is if the parsers installed by metron should *also* be
registered in the same way, such that the parser extension ui lists all of
the parsers.
This would allow removal and installation of metron parsers installed by
the system. Also, following on we can customize the install to not install
everything. It may also be more simple concept wise.
That is the basic thing. So the question is if we want to go in this
direction.
The not so basic thing is to still deploy the extensions into the system as
packages, but not install them, such that you can add an extension from a
file *and also* from a ‘repository’ of
extensions. Down the line we can support local and remote repositories etc.
So the options are:
1. As it is now on the feature branch ( still pretending ;)), the system
installed parsers are not the same as the extension installed parsers.
While the project can uninstall and replace these parsers, the user/ui
cannot.
2. All parsers are installed as extensions and can be removed, but are all
initially installed
3. All parsers are extensions, but not installed during the original
deployment, rather they are put into a repository that the ui lets you
browse to select, in addition to allowing
install from file ( like intellij and other plugin systems do ). **This
would have implications for the demo system, since we would want to still
install the bro, snort and maybe yaf parsers.
In considering these questions, we need to keep in mind where we want to
go, and how much of this is required for first release of the extension
system. There is a lot of ‘while we are already doing xxx, we might as
well do yyyy since it will be harder later’ in this.
Thanks for your time and your timely responses.
ottO