You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by James Sirota <js...@apache.org> on 2017/09/21 15:06:15 UTC
[DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Hi Guys,
I'd like to propose a community meeting for this Tuesday, Sept.23 and focus on the strategy for getting Metron-777 and related PRs on Otto's feature branch merged into master. Otto, is there a way you can record a youtube of adding a custom via your feature branch by Tueday? I'd like to be able to review that and get everyone's feedback so that we can start formulating our strategy on getting it in.
I propose to have this meeting at Tuesday, Sept.23 10AM PST if that works for everyone:
https://hortonworks.webex.com/meet/jsirota
Call-in toll-free number (US/Canada) 1-877-668-4493
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Kyle Richardson <ky...@gmail.com>.
I've been stretched a little thin lately but am trying to jump back in.
2:30pm EDT works well for me too.
-Kyle
On Mon, Sep 25, 2017 at 8:38 PM, Otto Fowler <ot...@gmail.com>
wrote:
> 11:30 your time. Sorry I have to pick my kids up from school. 2:30 mine.
>
>
> On September 25, 2017 at 19:41:28, James Sirota (jsirota@apache.org)
> wrote:
>
> Oh sorry, didn't notice that. Otto, when is a good time for you?
>
> 25.09.2017, 16:35, "Zeolla@GMail.com" <ze...@gmail.com>:
> > When is the meeting, given Otto mentioned he can't make 10am? Or did that
> > change
> >
> > Jon
> >
> > On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
> >
> >> Great. Thank you, Otto. I would encourage everyone to watch it so that
> we
> >> have constructive feedback for tomorrow and are able to arrive to a
> decision
> >>
> >> Thanks,
> >> James
> >>
> >> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
> >> > https://youtu.be/-ISycoP3TVA
> >> >
> >> > The video is short and simple. Hopefully it is what you are looking
> for.
> >> >
> >> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com (
> zeolla@gmail.com)
>
> >> > wrote:
> >> >
> >> > I won't be able to make it and would really like to make sure there's
> a
> >> > recording for this one, if possible. I'm unavailable until Thursday
> of
> >> > next week, but not necessarily suggesting this gets moved.
> >> >
> >> > Jon
> >> >
> >> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com>
> wrote:
> >> >
> >> >> I can’t make that time, can we make it later in the day?
> >> >>
> >> >> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org
> )
> >> >> wrote:
> >> >>
> >> >> https://hortonworks.webex.com/meet/jsirota
> >> > --
> >> >
> >> > Jon
> >>
> >> -------------------
> >> Thank you,
> >>
> >> James Sirota
> >> PPMC- Apache Metron (Incubating)
> >> jsirota AT apache DOT org
> > --
> >
> > Jon
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Otto Fowler <ot...@gmail.com>.
I am having to request access, not sure if this is what you meant to do.
On October 3, 2017 at 18:56:29, James Sirota (jsirota@apache.org) wrote:
Link to the recording. You need a WebEx player to view it
https://drive.google.com/open?id=0B3a8U3GCzkKNZHR2LUZNcmljcTQ
26.09.2017, 09:20, "Laurens Vets" <la...@daemon.be>:
> 11:30 won't work for me, but that's fine. I only had 1 comment on Otto's
> video: What happens when we have 2 parsers/sensors with the same name.
> If there's ever a parser/sensor repository, this might be an issue.
>
> On 2017-09-25 17:38, Otto Fowler wrote:
>> 11:30 your time. Sorry I have to pick my kids up from school. 2:30
>> mine.
>>
>> On September 25, 2017 at 19:41:28, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> Oh sorry, didn't notice that. Otto, when is a good time for you?
>>
>> 25.09.2017, 16:35, "Zeolla@GMail.com" <ze...@gmail.com>:
>>> When is the meeting, given Otto mentioned he can't make 10am? Or did
>>> that
>>> change
>>>
>>> Jon
>>>
>>> On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
>>>
>>>> Great. Thank you, Otto. I would encourage everyone to watch it so
>>>> that
>> we
>>>> have constructive feedback for tomorrow and are able to arrive to a
>> decision
>>>> Thanks,
>>>> James
>>>>
>>>> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
>>>> > https://youtu.be/-ISycoP3TVA
>>>> >
>>>> > The video is short and simple. Hopefully it is what you are
>>>> looking
>> for.
>>>> >
>>>> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com
>>>> (zeolla@gmail.com)
>>
>>>> > wrote:
>>>> >
>>>> > I won't be able to make it and would really like to make sure
>>>> there's
>> a
>>>> > recording for this one, if possible. I'm unavailable until
>>>> Thursday
>> of
>>>> > next week, but not necessarily suggesting this gets moved.
>>>> >
>>>> > Jon
>>>> >
>>>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com>
>> wrote:
>>>> >
>>>> >> I can’t make that time, can we make it later in the day?
>>>> >>
>>>> >> On September 21, 2017 at 11:40:37, James Sirota
>>>> (jsirota@apache.org)
>>>> >> wrote:
>>>> >>
>>>> >> https://hortonworks.webex.com/meet/jsirota
>>>> > --
>>>> >
>>>> > Jon
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PPMC- Apache Metron (Incubating)
>>>> jsirota AT apache DOT org
>>> --
>>>
>>> Jon
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by James Sirota <js...@apache.org>.
Link to the recording. You need a WebEx player to view it
https://drive.google.com/open?id=0B3a8U3GCzkKNZHR2LUZNcmljcTQ
26.09.2017, 09:20, "Laurens Vets" <la...@daemon.be>:
> 11:30 won't work for me, but that's fine. I only had 1 comment on Otto's
> video: What happens when we have 2 parsers/sensors with the same name.
> If there's ever a parser/sensor repository, this might be an issue.
>
> On 2017-09-25 17:38, Otto Fowler wrote:
>> 11:30 your time. Sorry I have to pick my kids up from school. 2:30
>> mine.
>>
>> On September 25, 2017 at 19:41:28, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> Oh sorry, didn't notice that. Otto, when is a good time for you?
>>
>> 25.09.2017, 16:35, "Zeolla@GMail.com" <ze...@gmail.com>:
>>> When is the meeting, given Otto mentioned he can't make 10am? Or did
>>> that
>>> change
>>>
>>> Jon
>>>
>>> On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
>>>
>>>> Great. Thank you, Otto. I would encourage everyone to watch it so
>>>> that
>> we
>>>> have constructive feedback for tomorrow and are able to arrive to a
>> decision
>>>> Thanks,
>>>> James
>>>>
>>>> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
>>>> > https://youtu.be/-ISycoP3TVA
>>>> >
>>>> > The video is short and simple. Hopefully it is what you are
>>>> looking
>> for.
>>>> >
>>>> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com
>>>> (zeolla@gmail.com)
>>
>>>> > wrote:
>>>> >
>>>> > I won't be able to make it and would really like to make sure
>>>> there's
>> a
>>>> > recording for this one, if possible. I'm unavailable until
>>>> Thursday
>> of
>>>> > next week, but not necessarily suggesting this gets moved.
>>>> >
>>>> > Jon
>>>> >
>>>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com>
>> wrote:
>>>> >
>>>> >> I can’t make that time, can we make it later in the day?
>>>> >>
>>>> >> On September 21, 2017 at 11:40:37, James Sirota
>>>> (jsirota@apache.org)
>>>> >> wrote:
>>>> >>
>>>> >> https://hortonworks.webex.com/meet/jsirota
>>>> > --
>>>> >
>>>> > Jon
>>>>
>>>> -------------------
>>>> Thank you,
>>>>
>>>> James Sirota
>>>> PPMC- Apache Metron (Incubating)
>>>> jsirota AT apache DOT org
>>> --
>>>
>>> Jon
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Laurens Vets <la...@daemon.be>.
11:30 won't work for me, but that's fine. I only had 1 comment on Otto's
video: What happens when we have 2 parsers/sensors with the same name.
If there's ever a parser/sensor repository, this might be an issue.
On 2017-09-25 17:38, Otto Fowler wrote:
> 11:30 your time. Sorry I have to pick my kids up from school. 2:30
> mine.
>
>
> On September 25, 2017 at 19:41:28, James Sirota (jsirota@apache.org)
> wrote:
>
> Oh sorry, didn't notice that. Otto, when is a good time for you?
>
> 25.09.2017, 16:35, "Zeolla@GMail.com" <ze...@gmail.com>:
>> When is the meeting, given Otto mentioned he can't make 10am? Or did
>> that
>> change
>>
>> Jon
>>
>> On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
>>
>>> Great. Thank you, Otto. I would encourage everyone to watch it so
>>> that
> we
>>> have constructive feedback for tomorrow and are able to arrive to a
> decision
>>>
>>> Thanks,
>>> James
>>>
>>> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
>>> > https://youtu.be/-ISycoP3TVA
>>> >
>>> > The video is short and simple. Hopefully it is what you are
>>> looking
> for.
>>> >
>>> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com
>>> (zeolla@gmail.com)
>
>>> > wrote:
>>> >
>>> > I won't be able to make it and would really like to make sure
>>> there's
> a
>>> > recording for this one, if possible. I'm unavailable until
>>> Thursday
> of
>>> > next week, but not necessarily suggesting this gets moved.
>>> >
>>> > Jon
>>> >
>>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com>
> wrote:
>>> >
>>> >> I can’t make that time, can we make it later in the day?
>>> >>
>>> >> On September 21, 2017 at 11:40:37, James Sirota
>>> (jsirota@apache.org)
>>> >> wrote:
>>> >>
>>> >> https://hortonworks.webex.com/meet/jsirota
>>> > --
>>> >
>>> > Jon
>>>
>>> -------------------
>>> Thank you,
>>>
>>> James Sirota
>>> PPMC- Apache Metron (Incubating)
>>> jsirota AT apache DOT org
>> --
>>
>> Jon
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Otto Fowler <ot...@gmail.com>.
11:30 your time. Sorry I have to pick my kids up from school. 2:30 mine.
On September 25, 2017 at 19:41:28, James Sirota (jsirota@apache.org) wrote:
Oh sorry, didn't notice that. Otto, when is a good time for you?
25.09.2017, 16:35, "Zeolla@GMail.com" <ze...@gmail.com>:
> When is the meeting, given Otto mentioned he can't make 10am? Or did that
> change
>
> Jon
>
> On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
>
>> Great. Thank you, Otto. I would encourage everyone to watch it so that
we
>> have constructive feedback for tomorrow and are able to arrive to a
decision
>>
>> Thanks,
>> James
>>
>> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
>> > https://youtu.be/-ISycoP3TVA
>> >
>> > The video is short and simple. Hopefully it is what you are looking
for.
>> >
>> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
>> > wrote:
>> >
>> > I won't be able to make it and would really like to make sure there's
a
>> > recording for this one, if possible. I'm unavailable until Thursday
of
>> > next week, but not necessarily suggesting this gets moved.
>> >
>> > Jon
>> >
>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com>
wrote:
>> >
>> >> I can’t make that time, can we make it later in the day?
>> >>
>> >> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
>> >> wrote:
>> >>
>> >> https://hortonworks.webex.com/meet/jsirota
>> > --
>> >
>> > Jon
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
> --
>
> Jon
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by James Sirota <js...@apache.org>.
Oh sorry, didn't notice that. Otto, when is a good time for you?
25.09.2017, 16:35, "Zeolla@GMail.com" <ze...@gmail.com>:
> When is the meeting, given Otto mentioned he can't make 10am? Or did that
> change
>
> Jon
>
> On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
>
>> Great. Thank you, Otto. I would encourage everyone to watch it so that we
>> have constructive feedback for tomorrow and are able to arrive to a decision
>>
>> Thanks,
>> James
>>
>> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
>> > https://youtu.be/-ISycoP3TVA
>> >
>> > The video is short and simple. Hopefully it is what you are looking for.
>> >
>> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
>> > wrote:
>> >
>> > I won't be able to make it and would really like to make sure there's a
>> > recording for this one, if possible. I'm unavailable until Thursday of
>> > next week, but not necessarily suggesting this gets moved.
>> >
>> > Jon
>> >
>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
>> >
>> >> I can’t make that time, can we make it later in the day?
>> >>
>> >> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
>> >> wrote:
>> >>
>> >> https://hortonworks.webex.com/meet/jsirota
>> > --
>> >
>> > Jon
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org
> --
>
> Jon
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
When is the meeting, given Otto mentioned he can't make 10am? Or did that
change
Jon
On Mon, Sep 25, 2017, 19:19 James Sirota <js...@apache.org> wrote:
> Great. Thank you, Otto. I would encourage everyone to watch it so that we
> have constructive feedback for tomorrow and are able to arrive to a decision
>
> Thanks,
> James
>
> 25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
> > https://youtu.be/-ISycoP3TVA
> >
> > The video is short and simple. Hopefully it is what you are looking for.
> >
> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
> > wrote:
> >
> > I won't be able to make it and would really like to make sure there's a
> > recording for this one, if possible. I'm unavailable until Thursday of
> > next week, but not necessarily suggesting this gets moved.
> >
> > Jon
> >
> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
> >
> >> I can’t make that time, can we make it later in the day?
> >>
> >> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
> >> wrote:
> >>
> >> https://hortonworks.webex.com/meet/jsirota
> > --
> >
> > Jon
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
--
Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by James Sirota <js...@apache.org>.
Great. Thank you, Otto. I would encourage everyone to watch it so that we have constructive feedback for tomorrow and are able to arrive to a decision
Thanks,
James
25.09.2017, 08:27, "Otto Fowler" <ot...@gmail.com>:
> https://youtu.be/-ISycoP3TVA
>
> The video is short and simple. Hopefully it is what you are looking for.
>
> On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> I won't be able to make it and would really like to make sure there's a
> recording for this one, if possible. I'm unavailable until Thursday of
> next week, but not necessarily suggesting this gets moved.
>
> Jon
>
> On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
>
>> I can’t make that time, can we make it later in the day?
>>
>> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> https://hortonworks.webex.com/meet/jsirota
> --
>
> Jon
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Otto Fowler <ot...@gmail.com>.
Great!
Since it looks like I’ll be restating and refactoring things on master in
chunks this is still a ways off. When I get to the ui/rest area I will
refactor to this. When that time comes, I would like for us to talk about
having the concept of a local repository.
On top of what we just talked about….
We would install extensions to the repository in hdfs. When you went to
install an extension, you could upload or pick from the repository. We
would then get into the flow we were just talking about. So the user would
only install the parser extensions they need. Also, we could then
introduce remote repositories etc.
Another question is signing extensions. We will have to think about that
before landing.
On September 26, 2017 at 07:06:23, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Hi Otto,
Sounds like we’re really getting somewhere there. I really like that idea.
If we have configs as a kind of default, which we can then use to bootstrap
any new sensor based on a parser from the bundles, that gives us the best
of every world.
Also, I’d agree with you that we should move all the parsers to the bundle
method (as you have in the code base) and that the UI should therefore
treat the parsers we supply as a project (CEF, bro, snort etc) as no
different from a parser an end user writes and bundles. The only area I
would say we want to distinguish, as you already have I believe, is the
base parsers (Java, Grok) since those would be a start point for
inheritance.
If we can get the configs as default bootstrap rather than source of truth,
I think we will have a great system. Let me know if there is anything
around that I can help with on the implementation side.
Simon
On 26 Sep 2017, at 11:53, Otto Fowler <ot...@gmail.com> wrote:
Hi Simon
1. At this moment, in the unreviewed PR ( although this kind of counts ;)
) it will overwrite.
2. You just create a new sensor, based on that type and give it a different
name, same as you would now. You would have to start each configuration
from nothing however, because the UI doesn’t copy an existing sensor -or-
copy from the extension default ( where they exist )
More on 1. I have mentioned in the PR and other threads about these
changes being the start of a lifecycle for parsers/sensors. The idea being
that you install the parsers, and their default configurations but doesn’t
create the sensor by default as we do now ( both in master and fb ), and
then when you create a parser, it copies the default configuration, using
that to pre- populate the configuration ui. Thus, the install would never
touch the deployed configurations. I did not go so far as to implement
that, but I think that is where we can go. Short of that, I can change the
extensions to not install default configs or not to overwrite if they
exist. BTW. The issue you are speaking about I believe is present in
metron in general before this change. If someone could clarify the upgrade
stuff in ambari it would help. We don’t check if there are already
configurations in zookeeper ever before pushing configurations out.
This also brings me back to my discuss question from the other day: The
extension list you see in the demo, only has the user installed parser
extensions, not the system ( metron ) ones, like bro, yaf, snort etc. Thus
they are not managed the same way, and they are not registered the same way
with the default configurations in a separate place.
I think what we want is all parsers installed as extensions, not to install
the parsers as sensors by default as we do now, and the configuration ui to
use the default from the extension registration when creating new. I think
the ‘demo’ system should be a separate ambari install all together. IE> The
install of bro, snort, the dashboard, es templates.
We are now a long ways off from landing the UI pr. I was always expecting
refactoring around the configuration management and install part, so
please, let me know what you think and I can take a shot at it. Don’t
forget about irc.
On September 26, 2017 at 05:17:32, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Hi Otto,
This is a great demo, nice and clear, many thanks.
Two questions remain for me:
1. how I would change configuration outside of the bundle? i.e. I install a
bundle and that gives me enrichment and indexing config, but I then want to
tune indexing for the characteristics of the actual sensor, or I want to
add use cases in enrichment which are customised to my org, what happens if
I update the bundle, does it overwrite the zookeeper config changes? Do I
have to add all my enrichment config in the bundle? That would be an
absolute blocker for SOC ops users who do not have access to maven for
example and would render the management ui null and void.
2. At the moment it seems bundle implies sensor. How would I have a parser
(e.g. CEF parser) which was used against multiple topics (i.e. one parser
bundle for a data type, multiple sensors based on multiple topics feeding
that type)?
Really keen to get this in as an extension mechanism, as I think it will be
a huge help to people developing custom parsers, but I really want to make
sure it will not clobber the use cases people add with enrichment config in
particular.
Simon
> On 25 Sep 2017, at 16:27, Otto Fowler <ot...@gmail.com> wrote:
>
> https://youtu.be/-ISycoP3TVA
>
> The video is short and simple. Hopefully it is what you are looking for.
>
>
> On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> I won't be able to make it and would really like to make sure there's a
> recording for this one, if possible. I'm unavailable until Thursday of
> next week, but not necessarily suggesting this gets moved.
>
> Jon
>
> On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
>
>> I can’t make that time, can we make it later in the day?
>>
>>
>> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> https://hortonworks.webex.com/meet/jsirota
>>
> --
>
> Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Hi Otto,
Sounds like we’re really getting somewhere there. I really like that idea. If we have configs as a kind of default, which we can then use to bootstrap any new sensor based on a parser from the bundles, that gives us the best of every world.
Also, I’d agree with you that we should move all the parsers to the bundle method (as you have in the code base) and that the UI should therefore treat the parsers we supply as a project (CEF, bro, snort etc) as no different from a parser an end user writes and bundles. The only area I would say we want to distinguish, as you already have I believe, is the base parsers (Java, Grok) since those would be a start point for inheritance.
If we can get the configs as default bootstrap rather than source of truth, I think we will have a great system. Let me know if there is anything around that I can help with on the implementation side.
Simon
> On 26 Sep 2017, at 11:53, Otto Fowler <ot...@gmail.com> wrote:
>
> Hi Simon
>
> 1. At this moment, in the unreviewed PR ( although this kind of counts ;) ) it will overwrite.
> 2. You just create a new sensor, based on that type and give it a different name, same as you would now. You would have to start each configuration from nothing however, because the UI doesn’t copy an existing sensor -or- copy from the extension default ( where they exist )
>
> More on 1. I have mentioned in the PR and other threads about these changes being the start of a lifecycle for parsers/sensors. The idea being that you install the parsers, and their default configurations but doesn’t create the sensor by default as we do now ( both in master and fb ), and then when you create a parser, it copies the default configuration, using that to pre- populate the configuration ui. Thus, the install would never touch the deployed configurations. I did not go so far as to implement that, but I think that is where we can go. Short of that, I can change the extensions to not install default configs or not to overwrite if they exist. BTW. The issue you are speaking about I believe is present in metron in general before this change. If someone could clarify the upgrade stuff in ambari it would help. We don’t check if there are already configurations in zookeeper ever before pushing configurations out.
>
> This also brings me back to my discuss question from the other day: The extension list you see in the demo, only has the user installed parser extensions, not the system ( metron ) ones, like bro, yaf, snort etc. Thus they are not managed the same way, and they are not registered the same way with the default configurations in a separate place.
>
> I think what we want is all parsers installed as extensions, not to install the parsers as sensors by default as we do now, and the configuration ui to use the default from the extension registration when creating new. I think the ‘demo’ system should be a separate ambari install all together. IE> The install of bro, snort, the dashboard, es templates.
>
> We are now a long ways off from landing the UI pr. I was always expecting refactoring around the configuration management and install part, so please, let me know what you think and I can take a shot at it. Don’t forget about irc.
>
>
>
>
> On September 26, 2017 at 05:17:32, Simon Elliston Ball (simon@simonellistonball.com <ma...@simonellistonball.com>) wrote:
>
>> Hi Otto,
>>
>> This is a great demo, nice and clear, many thanks.
>>
>> Two questions remain for me:
>>
>> 1. how I would change configuration outside of the bundle? i.e. I install a bundle and that gives me enrichment and indexing config, but I then want to tune indexing for the characteristics of the actual sensor, or I want to add use cases in enrichment which are customised to my org, what happens if I update the bundle, does it overwrite the zookeeper config changes? Do I have to add all my enrichment config in the bundle? That would be an absolute blocker for SOC ops users who do not have access to maven for example and would render the management ui null and void.
>> 2. At the moment it seems bundle implies sensor. How would I have a parser (e.g. CEF parser) which was used against multiple topics (i.e. one parser bundle for a data type, multiple sensors based on multiple topics feeding that type)?
>>
>> Really keen to get this in as an extension mechanism, as I think it will be a huge help to people developing custom parsers, but I really want to make sure it will not clobber the use cases people add with enrichment config in particular.
>>
>> Simon
>>
>> > On 25 Sep 2017, at 16:27, Otto Fowler <ottobackwards@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > https://youtu.be/-ISycoP3TVA <https://youtu.be/-ISycoP3TVA>
>> >
>> > The video is short and simple. Hopefully it is what you are looking for.
>> >
>> >
>> > On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com <ma...@gmail.com>)
>> > wrote:
>> >
>> > I won't be able to make it and would really like to make sure there's a
>> > recording for this one, if possible. I'm unavailable until Thursday of
>> > next week, but not necessarily suggesting this gets moved.
>> >
>> > Jon
>> >
>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwards@gmail.com <ma...@gmail.com>> wrote:
>> >
>> >> I can’t make that time, can we make it later in the day?
>> >>
>> >>
>> >> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org <ma...@apache.org>)
>> >> wrote:
>> >>
>> >> https://hortonworks.webex.com/meet/jsirota <https://hortonworks.webex.com/meet/jsirota>
>> >>
>> > --
>> >
>> > Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Otto Fowler <ot...@gmail.com>.
Hi Simon
1. At this moment, in the unreviewed PR ( although this kind of counts ;)
) it will overwrite.
2. You just create a new sensor, based on that type and give it a different
name, same as you would now. You would have to start each configuration
from nothing however, because the UI doesn’t copy an existing sensor -or-
copy from the extension default ( where they exist )
More on 1. I have mentioned in the PR and other threads about these
changes being the start of a lifecycle for parsers/sensors. The idea being
that you install the parsers, and their default configurations but doesn’t
create the sensor by default as we do now ( both in master and fb ), and
then when you create a parser, it copies the default configuration, using
that to pre- populate the configuration ui. Thus, the install would never
touch the deployed configurations. I did not go so far as to implement
that, but I think that is where we can go. Short of that, I can change the
extensions to not install default configs or not to overwrite if they
exist. BTW. The issue you are speaking about I believe is present in
metron in general before this change. If someone could clarify the upgrade
stuff in ambari it would help. We don’t check if there are already
configurations in zookeeper ever before pushing configurations out.
This also brings me back to my discuss question from the other day: The
extension list you see in the demo, only has the user installed parser
extensions, not the system ( metron ) ones, like bro, yaf, snort etc. Thus
they are not managed the same way, and they are not registered the same way
with the default configurations in a separate place.
I think what we want is all parsers installed as extensions, not to install
the parsers as sensors by default as we do now, and the configuration ui to
use the default from the extension registration when creating new. I think
the ‘demo’ system should be a separate ambari install all together. IE> The
install of bro, snort, the dashboard, es templates.
We are now a long ways off from landing the UI pr. I was always expecting
refactoring around the configuration management and install part, so
please, let me know what you think and I can take a shot at it. Don’t
forget about irc.
On September 26, 2017 at 05:17:32, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Hi Otto,
This is a great demo, nice and clear, many thanks.
Two questions remain for me:
1. how I would change configuration outside of the bundle? i.e. I install a
bundle and that gives me enrichment and indexing config, but I then want to
tune indexing for the characteristics of the actual sensor, or I want to
add use cases in enrichment which are customised to my org, what happens if
I update the bundle, does it overwrite the zookeeper config changes? Do I
have to add all my enrichment config in the bundle? That would be an
absolute blocker for SOC ops users who do not have access to maven for
example and would render the management ui null and void.
2. At the moment it seems bundle implies sensor. How would I have a parser
(e.g. CEF parser) which was used against multiple topics (i.e. one parser
bundle for a data type, multiple sensors based on multiple topics feeding
that type)?
Really keen to get this in as an extension mechanism, as I think it will be
a huge help to people developing custom parsers, but I really want to make
sure it will not clobber the use cases people add with enrichment config in
particular.
Simon
> On 25 Sep 2017, at 16:27, Otto Fowler <ot...@gmail.com> wrote:
>
> https://youtu.be/-ISycoP3TVA
>
> The video is short and simple. Hopefully it is what you are looking for.
>
>
> On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> I won't be able to make it and would really like to make sure there's a
> recording for this one, if possible. I'm unavailable until Thursday of
> next week, but not necessarily suggesting this gets moved.
>
> Jon
>
> On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
>
>> I can’t make that time, can we make it later in the day?
>>
>>
>> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> https://hortonworks.webex.com/meet/jsirota
>>
> --
>
> Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Hi Otto,
This is a great demo, nice and clear, many thanks.
Two questions remain for me:
1. how I would change configuration outside of the bundle? i.e. I install a bundle and that gives me enrichment and indexing config, but I then want to tune indexing for the characteristics of the actual sensor, or I want to add use cases in enrichment which are customised to my org, what happens if I update the bundle, does it overwrite the zookeeper config changes? Do I have to add all my enrichment config in the bundle? That would be an absolute blocker for SOC ops users who do not have access to maven for example and would render the management ui null and void.
2. At the moment it seems bundle implies sensor. How would I have a parser (e.g. CEF parser) which was used against multiple topics (i.e. one parser bundle for a data type, multiple sensors based on multiple topics feeding that type)?
Really keen to get this in as an extension mechanism, as I think it will be a huge help to people developing custom parsers, but I really want to make sure it will not clobber the use cases people add with enrichment config in particular.
Simon
> On 25 Sep 2017, at 16:27, Otto Fowler <ot...@gmail.com> wrote:
>
> https://youtu.be/-ISycoP3TVA
>
> The video is short and simple. Hopefully it is what you are looking for.
>
>
> On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> I won't be able to make it and would really like to make sure there's a
> recording for this one, if possible. I'm unavailable until Thursday of
> next week, but not necessarily suggesting this gets moved.
>
> Jon
>
> On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
>
>> I can’t make that time, can we make it later in the day?
>>
>>
>> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
>> wrote:
>>
>> https://hortonworks.webex.com/meet/jsirota
>>
> --
>
> Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Otto Fowler <ot...@gmail.com>.
https://youtu.be/-ISycoP3TVA
The video is short and simple. Hopefully it is what you are looking for.
On September 21, 2017 at 16:54:13, Zeolla@GMail.com (zeolla@gmail.com)
wrote:
I won't be able to make it and would really like to make sure there's a
recording for this one, if possible. I'm unavailable until Thursday of
next week, but not necessarily suggesting this gets moved.
Jon
On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
> I can’t make that time, can we make it later in the day?
>
>
> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
> wrote:
>
> https://hortonworks.webex.com/meet/jsirota
>
--
Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I won't be able to make it and would really like to make sure there's a
recording for this one, if possible. I'm unavailable until Thursday of
next week, but not necessarily suggesting this gets moved.
Jon
On Thu, Sep 21, 2017, 15:04 Otto Fowler <ot...@gmail.com> wrote:
> I can’t make that time, can we make it later in the day?
>
>
> On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org)
> wrote:
>
> https://hortonworks.webex.com/meet/jsirota
>
--
Jon
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by Otto Fowler <ot...@gmail.com>.
I can’t make that time, can we make it later in the day?
On September 21, 2017 at 11:40:37, James Sirota (jsirota@apache.org) wrote:
https://hortonworks.webex.com/meet/jsirota
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Posted by James Sirota <js...@apache.org>.
Sorry made a mistake. Tuesday is 26th, not 23rd.
Thanks,
James
21.09.2017, 08:06, "James Sirota" <js...@apache.org>:
> Hi Guys,
>
> I'd like to propose a community meeting for this Tuesday, Sept.23 and focus on the strategy for getting Metron-777 and related PRs on Otto's feature branch merged into master. Otto, is there a way you can record a youtube of adding a custom via your feature branch by Tueday? I'd like to be able to review that and get everyone's feedback so that we can start formulating our strategy on getting it in.
>
> I propose to have this meeting at Tuesday, Sept.23 10AM PST if that works for everyone:
>
> https://hortonworks.webex.com/meet/jsirota
> Call-in toll-free number (US/Canada) 1-877-668-4493
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org