You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by "Zeolla@GMail.com" <ze...@gmail.com> on 2017/03/31 01:09:38 UTC
[DISCUSS] The bro kafka plugin
Today I was taking a look at METRON-812
<https://issues.apache.org/jira/browse/METRON-812>, which made me recall
some conversations from a while back regarding where the bro kafka plugin
should ultimately live, and how to update it.
Back in METRON-348 <https://issues.apache.org/jira/browse/METRON-348> I
brought up the fact that some important changes
<https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a8353b4a0a8>
were made to the externally hosted version of the kafka plugin, and were
never introduced to Metron's hosted version (i.e. the one we use
<https://github.com/apache/incubator-metron/blob/master/metron-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
in vagrant when bro is installed). The conversation went down the route of
discussing whether or not the bro kafka plugin code should continue to live
in Metron in the first place. Now, with METRON-812, I see us further
muddying the waters of where to go for the right plugin, as our version is
still missing the public changes but adds some very important new
functionality.
I'd like to bring up the idea of using bro's packages
<https://github.com/bro/packages> framework, released in late 2016
<http://blog.bro.org/2016/10/introducing-bro-package-manager.html> (additional
documentation here <http://bro-package-manager.readthedocs.io/en/stable/>),
as a potential place for this to be hosted/referenced. This is a simple
and supported method (funded by Mozilla
<https://blog.mozilla.org/blog/2015/12/10/mozilla-open-source-support-first-awards-made/>)
to install and uninstall bro scripts, plugins, etc., and it also allows us
to continue to have enough control over updates to the plugin so that it
will not slow down Metron development by having it as a dependency
(resolving both of Casey's concerns noted here
<https://issues.apache.org/jira/browse/METRON-348?focusedCommentId=15391865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15391865>,
and I think this solution is supported by Nick's comments here
<https://issues.apache.org/jira/browse/METRON-348?focusedCommentId=15391872&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15391872>
as
well).
The only thing I'm not sure about is where to host the plugin itself - my
first thought would be Nick's github <https://github.com/nickwallen>, as he
really kicked off this effort, but maybe we can think of something better.
Is this approach of interest to anybody? It is extremely simple to put
together - I was able to throw one together
<https://github.com/bro/packages/blob/master/jonzeolla/bro-pkg.index> and
get it working with a fresh bro 2.5 install when attending the brocon talk
<https://www.bro.org/brocon2016/brocon2016_abstracts.html#bro-packagemanager>
(recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
<https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that introduced this
to me in the first place.
Jon
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Casey Stella <ce...@gmail.com>.
fine by me.
On Wed, Apr 5, 2017 at 4:16 PM, James Sirota <js...@apache.org> wrote:
> I don't mind incubator-metron-bro-plugin-kafka
>
> 05.04.2017, 13:15, "Nick Allen" <ni...@nickallen.org>:
> > Ok, I found that CouchDB and Nifi, among others, use multiple repos. A
> > fairly obvious pattern seems to emerge. :)
> >
> > git://git.apache.org/couchdb-fabric.git
> > git://git.apache.org/couchdb-couch-replicator.git
> > git://git.apache.org/couchdb-chttpd.git
> >
> > git://git.apache.org/nifi
> > git://git.apache.org/nifi-site.git
> >
> >
> >
> > So what shall we call the new repo? Using
> > "incubator-metron-bro-plugin-kafka" is the obvious choice, but seems
> > excessively long. Totally open to better name for the plugin itself.
> >
> > On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <ni...@nickallen.org> wrote:
> >
> >> Does anyone know any other Apache projects that are using multiple
> repos?
> >> I'd like to see what they've done just so we don't break convention.
> >>
> >> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
> >>
> >>> Yes, I will open an INFRA ticket. Just give me a little time to
> research
> >>> what we need.
> >>>
> >>> On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com>
> >>> wrote:
> >>>
> >>>> Okay great, thanks. Would you mind throwing in an INFRA ticket for
> the
> >>>> new
> >>>> repo? I can take it all from there.
> >>>>
> >>>> Does anybody know if we have ASF resources to help answer the above
> legal
> >>>> question?
> >>>>
> >>>> Jon
> >>>>
> >>>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org>
> wrote:
> >>>>
> >>>> > (1) I am not sure if licensing is a problem here.
> >>>> >
> >>>> > (2) I am OK with whatever we need to get this effort done and under
> >>>> ASF.
> >>>> >
> >>>> >
> >>>> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> >>>> wrote:
> >>>> >
> >>>> > > I'm working on this
> >>>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348>
> in
> >>>> > > preparation for the new repo and migration to a package. It looks
> >>>> like
> >>>> > in
> >>>> > > bro-plugins COPYING
> >>>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
> >>>> > attributes
> >>>> > > to Nick, but in our version COPYING
> >>>> > > <https://github.com/apache/incubator-metron/tree/master/
> >>>> > > metron-sensors/bro-plugin-kafka/COPYING>
> >>>> > > points to the Apache License. Same with MAINTAINER (this
> >>>> > > <https://github.com/apache/incubator-metron/blob/master/
> >>>> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> >>>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/
> MAINTA
> >>>> INER
> >>>> > > >).
> >>>> > > I assume when we package this up and host it in Apache we need to
> >>>> give it
> >>>> > > the Apache license, and point to Metron for MAINTAINER. My
> questions
> >>>> > are:
> >>>> > >
> >>>> > > 1. Is there any legal/licensing concern here? I am taking changes
> >>>> from
> >>>> > > the bro-plugins version and pulling it into the Apache-hosted
> code.
> >>>> > IANAL
> >>>> > > 2. Nick - are you OK with these changes?
> >>>> > >
> >>>> > > Jon
> >>>> > >
> >>>> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> zeolla@gmail.com>
> >>>> > wrote:
> >>>> > >
> >>>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like
> >>>> > > > <https://www.apache.org/dev/infra-contact> committers aren't
> >>>> supposed
> >>>> > > to.
> >>>> > > >
> >>>> > > > Jon
> >>>> > > >
> >>>> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> zeolla@gmail.com
> >>>> >
> >>>> > > wrote:
> >>>> > > >
> >>>> > > > I would be happy to try it again but I attempted to do that
> before
> >>>> with
> >>>> > > > bro packages and it failed to be able to handle it. I also
> tried
> >>>> using
> >>>> > > > branches of a repo with bro but that similarly failed (and was
> a
> >>>> pretty
> >>>> > > bad
> >>>> > > > idea to start with).
> >>>> > > >
> >>>> > > > Jon
> >>>> > > >
> >>>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org>
> wrote:
> >>>> > > >
> >>>> > > > We should be able to request just one alternate repo from
> INFRA,
> >>>> and
> >>>> > put
> >>>> > > a
> >>>> > > > top hierarchical level in it that doesn’t include a maven pom.
> As
> >>>> far
> >>>> > as
> >>>> > > > maven and clients are concerned, it
> >>>> > > >
> >>>> > > > just increases by 1 the path length to the root of the repo.
> >>>> > > >
> >>>> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com>
> wrote:
> >>>> > > >
> >>>> > > > Once we agree on a repo location to host this, I would be
> >>>> happy to
> >>>> > > put
> >>>> > > > together the package and update our environments to use
> >>>> bro-pkg to
> >>>> > > > install
> >>>> > > > the plugin. I have created METRON-813
> >>>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track
> >>>> this
> >>>> > and
> >>>> > > > changed METRON-348 <
> >>>> > https://issues.apache.org/jira/browse/METRON-348
> >>>> > > >
> >>>> > > > to be
> >>>> > > > a sub-task.
> >>>> > > >
> >>>> > > > Otto - the bro packages model doesn't allow colocation with
> >>>> > anything
> >>>> > > > else.
> >>>> > > > That said, if we have two similar situations, and given the
> >>>> INFRA
> >>>> > > > example
> >>>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
> >>>> linked to
> >>>> > > > before
> >>>> > > > was requesting 9 repos, perhaps we just request two repos.
> >>>> Would
> >>>> > > > someone
> >>>> > > > else mind putting that request in?
> >>>> > > >
> >>>> > > > Jon
> >>>> > > >
> >>>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> >>>> > > ottobackwards@gmail.com>
> >>>> > > > wrote:
> >>>> > > >
> >>>> > > > Could we create a separate repo for more than on thing? like
> >>>> put …
> >>>> > > um
> >>>> > > > let’s say
> >>>> > > > a maven plugin and the bro plugin?
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
> >>>> > > wrote:
> >>>> > > >
> >>>> > > > I agree with everything that I've read.
> >>>> > > >
> >>>> > > > One of the guys from Bro had contacted me a while back,
> >>>> letting me
> >>>> > > know
> >>>> > > > that the packaging mechanism in Bro was ready for public
> >>>> > > consumption. I
> >>>> > > > just have not had cycles to do anything with it yet. They are
> >>>> not
> >>>> > > > wanting
> >>>> > > > to host any of the plugins.
> >>>> > > >
> >>>> > > > I thought the package mechanism requires that a package live
> >>>> within
> >>>> > > > its own
> >>>> > > > repo (which Casey confirmed). This put me in a bind on how to
> >>>> > tackle
> >>>> > > > this. I don't want to personally host the plugin in my own
> >>>> Github
> >>>> > > > repo. I
> >>>> > > > would prefer that we host it in a community repo; either Bro or
> >>>> > > Metron.
> >>>> > > > Since Bro is moving away from hosting their own plugins, that
> >>>> > leaves
> >>>> > > > Metron.
> >>>> > > >
> >>>> > > > It would be great if we could create a separate repo for the
> >>>> > plugin.
> >>>> > > > That
> >>>> > > > solves the challenge of using the packaging mechanism.
> >>>> > > >
> >>>> > > > We do need to reconcile what is in bro/bro-plugins and what is
> >>>> in
> >>>> > > > Metron.
> >>>> > > > There are some enhancements that I and others have made that
> >>>> never
> >>>> > > > made it
> >>>> > > > back into Metron. They never made it back, because the original
> >>>> > plan
> >>>> > > > was
> >>>> > > > just to switch to using the plugin from bro/bro-plugins before
> >>>> the
> >>>> > > > idea of
> >>>> > > > a packaging mechanism hit Bro. Reconciling should be fairly
> >>>> easy to
> >>>> > > > see by
> >>>> > > > just doing a diff.
> >>>> > > >
> >>>> > > > It would be great if others want to take on any of that work. I
> >>>> > would
> >>>> > > > be
> >>>> > > > glad to offer any support that you need. Thanks, Jon!
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> >>>> > zeolla@gmail.com
> >>>> > > >
> >>>> > > > wrote:
> >>>> > > >
> >>>> > > > > Ok, great.
> >>>> > > > >
> >>>> > > > > I agree, I definitely want to hear from Nick on the topic. My
> >>>> > team
> >>>> > > is
> >>>> > > > > currently looking into enhancing the plugin as well to
> >>>> > potentially
> >>>> > > > allow
> >>>> > > > > sending to multiple clusters, investigating some issues we
> >>>> see
> >>>> > when
> >>>> > > > our
> >>>> > > > bro
> >>>> > > > > cluster is under load, turn it into a package, etc.
> >>>> > > > >
> >>>> > > > > The work you just did was on our to do list as well so I'm
> >>>> very
> >>>> > > > excited
> >>>> > > > to
> >>>> > > > > see it come through.
> >>>> > > > >
> >>>> > > > > Jon
> >>>> > > > >
> >>>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> >>>> cestella@gmail.com>
> >>>> > > > wrote:
> >>>> > > > >
> >>>> > > > > I *think* it's possible. People do ask for mirrors of
> >>>> directories
> >>>> > > > from
> >>>> > > > > time to time (see
> >>>> > https://issues.apache.org/jira/browse/INFRA-7060
> >>>> > > ).
> >>>> > > > If
> >>>> > > > > we
> >>>> > > > > think this is a good idea, we can pose it to INFRA as a
> >>>> request.
> >>>> > > I'd
> >>>> > > > love
> >>>> > > > > to see us be able to use the bro packaging infrastructure
> >>>> and get
> >>>> > > > more
> >>>> > > > > visibility for the plugin.
> >>>> > > > >
> >>>> > > > > I'd be particularly interested in Nick's opinion on this,
> >>>> though.
> >>>> > > > >
> >>>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> >>>> > > zeolla@gmail.com
> >>>> > > > >
> >>>> > > > > wrote:
> >>>> > > > >
> >>>> > > > > > You can version packages -
> >>>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> >>>> > > > > package.html#package-
> >>>> > > > > > versioning
> >>>> > > > > >
> >>>> > > > > > I agree that having a separate repo provided by Apache
> >>>> would be
> >>>> > > > optimal,
> >>>> > > > > I
> >>>> > > > > > just don't know the process for that or if it was even
> >>>> > reasonable
> >>>> > > > to
> >>>> > > > > > suggest.
> >>>> > > > > >
> >>>> > > > > > Jon
> >>>> > > > > >
> >>>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> >>>> > cestella@gmail.com>
> >>>> > > > wrote:
> >>>> > > > > >
> >>>> > > > > > > Looking at the bro packages, it appears that bro is
> >>>> expecting
> >>>> > > > things
> >>>> > > > to
> >>>> > > > > > be
> >>>> > > > > > > its own git repository. I wonder if we could either
> >>>> request
> >>>> > > INFRA
> >>>> > > > > > provide
> >>>> > > > > > > another repo for the bro-kafka plugin and integrate it
> >>>> into
> >>>> > > > metron as
> >>>> > > > a
> >>>> > > > > > git
> >>>> > > > > > > submodule *or* if we could request INFRA to create a
> >>>> github
> >>>> > > > mirror of
> >>>> > > > > the
> >>>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
> >>>> how
> >>>> > > > viable
> >>>> > > > > > either
> >>>> > > > > > > of those options are, frankly.
> >>>> > > > > > >
> >>>> > > > > > > One thing that I didn't see is how do you specify a
> >>>> > particular
> >>>> > > > release
> >>>> > > > > of
> >>>> > > > > > > the plugin that you want to install? For us, we'd want to
> >>>> > > > release the
> >>>> > > > > > > plugin along with the product. I didn't quite see how
> >>>> you'd
> >>>> > > push
> >>>> > > > > > releases
> >>>> > > > > > > for bro plugins.
> >>>> > > > > > >
> >>>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> >>>> > > > cestella@gmail.com>
> >>>> > > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > > So, I do agree with the concern. Is there a way to
> >>>> host the
> >>>> > > > package
> >>>> > > > > > > > within Metron? I definitely would like to see the
> >>>> > > > modifications at
> >>>> > > > > > > > https://github.com/bro/bro-plu
> >>>> gins/commit/b9f1f35415cb0db
> >>>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
> >>>> I'd
> >>>> > > > love for
> >>>> > > > > us
> >>>> > > > > > to
> >>>> > > > > > > > host the plugin.
> >>>> > > > > > > >
> >>>> > > > > > > > Thoughts?
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> >>>> > > > zeolla@gmail.com>
> >>>> > > >
> >>>> > > > > > > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > >> Today I was taking a look at METRON-812
> >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
> >>>> which
> >>>> > > > made me
> >>>> > > > > > > recall
> >>>> > > > > > > >> some conversations from a while back regarding where
> >>>> the
> >>>> > bro
> >>>> > > > kafka
> >>>> > > > > > > plugin
> >>>> > > > > > > >> should ultimately live, and how to update it.
> >>>> > > > > > > >>
> >>>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
> >>>> > > > > jira/browse/METRON-348>
> >>>> > > > > > I
> >>>> > > > > > > >> brought up the fact that some important changes
> >>>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> >>>> > > b9f1f35415cb0db06
> >>>> > > > > > > >> 5348da0a5043a8353b4a0a8>
> >>>> > > > > > > >> were made to the externally hosted version of the
> >>>> kafka
> >>>> > > > plugin,
> >>>> > > > and
> >>>> > > > > > were
> >>>> > > > > > > >> never introduced to Metron's hosted version (i.e. the
> >>>> one
> >>>> > we
> >>>> > > > use
> >>>> > > > > > > >> <https://github.com/apache/inc
> >>>> ubator-metron/blob/master/
> >>>> > > metr
> >>>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> >>>> > > > > > > >> in vagrant when bro is installed). The conversation
> >>>> went
> >>>> > > down
> >>>> > > > the
> >>>> > > > > > route
> >>>> > > > > > > >> of
> >>>> > > > > > > >> discussing whether or not the bro kafka plugin code
> >>>> should
> >>>> > > > continue
> >>>> > > > > to
> >>>> > > > > > > >> live
> >>>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I
> >>>> see
> >>>> > us
> >>>> > > > > further
> >>>> > > > > > > >> muddying the waters of where to go for the right
> >>>> plugin,
> >>>> > as
> >>>> > > > our
> >>>> > > > > > version
> >>>> > > > > > > is
> >>>> > > > > > > >> still missing the public changes but adds some very
> >>>> > > important
> >>>> > > > new
> >>>> > > > > > > >> functionality.
> >>>> > > > > > > >>
> >>>> > > > > > > >> I'd like to bring up the idea of using bro's packages
> >>>> > > > > > > >> <https://github.com/bro/packages> framework,
> >>>> released in
> >>>> > > > late 2016
> >>>> > > > > > > >> <
> >>>> > > > http://blog.bro.org/2016/10/introducing-bro-package-
> manager.html>
> >>>> > > > > > > >> (additional
> >>>> > > > > > > >> documentation here <
> >>>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> >>>> > > > > > > >> >),
> >>>> > > > > > > >> as a potential place for this to be hosted/referenced.
> >>>> > This
> >>>> > > > is a
> >>>> > > > > > simple
> >>>> > > > > > > >> and supported method (funded by Mozilla
> >>>> > > > > > > >> <https://blog.mozilla.org/blog
> >>>> /2015/12/10/mozilla-open-
> >>>> > > sourc
> >>>> > > > > > > >> e-support-first-awards-made/>)
> >>>> > > > > > > >> to install and uninstall bro scripts, plugins, etc.,
> >>>> and
> >>>> > it
> >>>> > > > also
> >>>> > > > > > allows
> >>>> > > > > > > us
> >>>> > > > > > > >> to continue to have enough control over updates to the
> >>>> > > plugin
> >>>> > > > so
> >>>> > > > > that
> >>>> > > > > > it
> >>>> > > > > > > >> will not slow down Metron development by having it as
> >>>> a
> >>>> > > > dependency
> >>>> > > > > > > >> (resolving both of Casey's concerns noted here
> >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> >>>> > > focusedCom
> >>>> > > > > > > >> mentId=15391865&page=com.atlas
> >>>> sian.jira.plugin.system.
> >>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> >>>> > > > > > > >> and I think this solution is supported by Nick's
> >>>> comments
> >>>> > > here
> >>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> >>>> > > focusedCom
> >>>> > > > > > > >> mentId=15391872&page=com.atlas
> >>>> sian.jira.plugin.system.
> >>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> >>>> > > > > > > >> as
> >>>> > > > > > > >> well).
> >>>> > > > > > > >>
> >>>> > > > > > > >> The only thing I'm not sure about is where to host the
> >>>> > > plugin
> >>>> > > > itself
> >>>> > > > > -
> >>>> > > > > > > my
> >>>> > > > > > > >> first thought would be Nick's github <
> >>>> > > > https://github.com/nickwallen
> >>>> > > > > >,
> >>>> > > > > > > as
> >>>> > > > > > > >> he
> >>>> > > > > > > >> really kicked off this effort, but maybe we can think
> >>>> of
> >>>> > > > something
> >>>> > > > > > > better.
> >>>> > > > > > > >>
> >>>> > > > > > > >> Is this approach of interest to anybody? It is
> >>>> extremely
> >>>> > > > simple to
> >>>> > > > > > put
> >>>> > > > > > > >> together - I was able to throw one together
> >>>> > > > > > > >> <https://github.com/bro/packages/blob/master/
> >>>> > > > > jonzeolla/bro-pkg.index>
> >>>> > > > > > > and
> >>>> > > > > > > >> get it working with a fresh bro 2.5 install when
> >>>> attending
> >>>> > > the
> >>>> > > > > brocon
> >>>> > > > > > > talk
> >>>> > > > > > > >> <
> >>>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> >>>> > > > > > > >> bro-packagemanager>
> >>>> > > > > > > >> (recording <https://www.youtube.com/watch
> >>>> ?v=9RFfPJeGkcE>,
> >>>> > > > slides
> >>>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> >>>> > that
> >>>> > > > > introduced
> >>>> > > > > > > >> this
> >>>> > > > > > > >> to me in the first place.
> >>>> > > > > > > >>
> >>>> > > > > > > >> Jon
> >>>> > > > > > > >> --
> >>>> > > > > > > >>
> >>>> > > > > > > >> Jon
> >>>> > > > > > > >>
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > >
> >>>> > > > > > --
> >>>> > > > > >
> >>>> > > > > > Jon
> >>>> > > > > >
> >>>> > > > >
> >>>> > > > > --
> >>>> > > > >
> >>>> > > > > Jon
> >>>> > > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > >
> >>>> > > > Jon
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > >
> >>>> > > > Jon
> >>>> > > >
> >>>> > > > --
> >>>> > > >
> >>>> > > > Jon
> >>>> > > >
> >>>> > > --
> >>>> > >
> >>>> > > Jon
> >>>> > >
> >>>> >
> >>>> --
> >>>>
> >>>> Jon
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
Re: [DISCUSS] The bro kafka plugin
Posted by James Sirota <js...@apache.org>.
I don't mind incubator-metron-bro-plugin-kafka
05.04.2017, 13:15, "Nick Allen" <ni...@nickallen.org>:
> Ok, I found that CouchDB and Nifi, among others, use multiple repos. A
> fairly obvious pattern seems to emerge. :)
>
> git://git.apache.org/couchdb-fabric.git
> git://git.apache.org/couchdb-couch-replicator.git
> git://git.apache.org/couchdb-chttpd.git
>
> git://git.apache.org/nifi
> git://git.apache.org/nifi-site.git
>
> \u200b
>
> So what shall we call the new repo? Using
> "incubator-metron-bro-plugin-kafka" is the obvious choice, but seems
> excessively long. Totally open to better name for the plugin itself.
>
> On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <ni...@nickallen.org> wrote:
>
>> �Does anyone know any other Apache projects that are using multiple repos?
>> �I'd like to see what they've done just so we don't break convention.
>>
>> �On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
>>
>>> �Yes, I will open an INFRA ticket. Just give me a little time to research
>>> �what we need.
>>>
>>> �On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com>
>>> �wrote:
>>>
>>>> �Okay great, thanks. Would you mind throwing in an INFRA ticket for the
>>>> �new
>>>> �repo? I can take it all from there.
>>>>
>>>> �Does anybody know if we have ASF resources to help answer the above legal
>>>> �question?
>>>>
>>>> �Jon
>>>>
>>>> �On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
>>>>
>>>> �> (1) I am not sure if licensing is a problem here.
>>>> �>
>>>> �> (2) I am OK with whatever we need to get this effort done and under
>>>> �ASF.
>>>> �>
>>>> �>
>>>> �> On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com>
>>>> �wrote:
>>>> �>
>>>> �> > I'm working on this
>>>> �> > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
>>>> �> > preparation for the new repo and migration to a package. It looks
>>>> �like
>>>> �> in
>>>> �> > bro-plugins COPYING
>>>> �> > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>>>> �> attributes
>>>> �> > to Nick, but in our version COPYING
>>>> �> > <https://github.com/apache/incubator-metron/tree/master/
>>>> �> > metron-sensors/bro-plugin-kafka/COPYING>
>>>> �> > points to the Apache License. Same with MAINTAINER (this
>>>> �> > <https://github.com/apache/incubator-metron/blob/master/
>>>> �> > metron-sensors/bro-plugin-kafka/MAINTAINER>
>>>> �> > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>>>> �INER
>>>> �> > >).
>>>> �> > I assume when we package this up and host it in Apache we need to
>>>> �give it
>>>> �> > the Apache license, and point to Metron for MAINTAINER. My questions
>>>> �> are:
>>>> �> >
>>>> �> > 1. Is there any legal/licensing concern here? I am taking changes
>>>> �from
>>>> �> > the bro-plugins version and pulling it into the Apache-hosted code.
>>>> �> IANAL
>>>> �> > 2. Nick - are you OK with these changes?
>>>> �> >
>>>> �> > Jon
>>>> �> >
>>>> �> > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
>>>> �> wrote:
>>>> �> >
>>>> �> > > Can someone on the PMC submit a ticket to INFRA? It looks like
>>>> �> > > <https://www.apache.org/dev/infra-contact> committers aren't
>>>> �supposed
>>>> �> > to.
>>>> �> > >
>>>> �> > > Jon
>>>> �> > >
>>>> �> > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <zeolla@gmail.com
>>>> �>
>>>> �> > wrote:
>>>> �> > >
>>>> �> > > I would be happy to try it again but I attempted to do that before
>>>> �with
>>>> �> > > bro packages and it failed to be able to handle it. I also tried
>>>> �using
>>>> �> > > branches of a repo with bro but that similarly failed (and was a
>>>> �pretty
>>>> �> > bad
>>>> �> > > idea to start with).
>>>> �> > >
>>>> �> > > Jon
>>>> �> > >
>>>> �> > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
>>>> �> > >
>>>> �> > > We should be able to request just one alternate repo from INFRA,
>>>> �and
>>>> �> put
>>>> �> > a
>>>> �> > > top hierarchical level in it that doesn\u2019t include a maven pom. As
>>>> �far
>>>> �> as
>>>> �> > > maven and clients are concerned, it
>>>> �> > >
>>>> �> > > just increases by 1 the path length to the root of the repo.
>>>> �> > >
>>>> �> > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>>> �> > >
>>>> �> > > Once we agree on a repo location to host this, I would be
>>>> �happy to
>>>> �> > put
>>>> �> > > together the package and update our environments to use
>>>> �bro-pkg to
>>>> �> > > install
>>>> �> > > the plugin. I have created METRON-813
>>>> �> > > <https://issues.apache.org/jira/browse/METRON-813> to track
>>>> �this
>>>> �> and
>>>> �> > > changed METRON-348 <
>>>> �> https://issues.apache.org/jira/browse/METRON-348
>>>> �> > >
>>>> �> > > to be
>>>> �> > > a sub-task.
>>>> �> > >
>>>> �> > > Otto - the bro packages model doesn't allow colocation with
>>>> �> anything
>>>> �> > > else.
>>>> �> > > That said, if we have two similar situations, and given the
>>>> �INFRA
>>>> �> > > example
>>>> �> > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
>>>> �linked to
>>>> �> > > before
>>>> �> > > was requesting 9 repos, perhaps we just request two repos.
>>>> �Would
>>>> �> > > someone
>>>> �> > > else mind putting that request in?
>>>> �> > >
>>>> �> > > Jon
>>>> �> > >
>>>> �> > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>>>> �> > ottobackwards@gmail.com>
>>>> �> > > wrote:
>>>> �> > >
>>>> �> > > Could we create a separate repo for more than on thing? like
>>>> �put \u2026
>>>> �> > um
>>>> �> > > let\u2019s say
>>>> �> > > a maven plugin and the bro plugin?
>>>> �> > >
>>>> �> > >
>>>> �> > >
>>>> �> > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
>>>> �> > wrote:
>>>> �> > >
>>>> �> > > I agree with everything that I've read.
>>>> �> > >
>>>> �> > > One of the guys from Bro had contacted me a while back,
>>>> �letting me
>>>> �> > know
>>>> �> > > that the packaging mechanism in Bro was ready for public
>>>> �> > consumption. I
>>>> �> > > just have not had cycles to do anything with it yet. They are
>>>> �not
>>>> �> > > wanting
>>>> �> > > to host any of the plugins.
>>>> �> > >
>>>> �> > > I thought the package mechanism requires that a package live
>>>> �within
>>>> �> > > its own
>>>> �> > > repo (which Casey confirmed). This put me in a bind on how to
>>>> �> tackle
>>>> �> > > this. I don't want to personally host the plugin in my own
>>>> �Github
>>>> �> > > repo. I
>>>> �> > > would prefer that we host it in a community repo; either Bro or
>>>> �> > Metron.
>>>> �> > > Since Bro is moving away from hosting their own plugins, that
>>>> �> leaves
>>>> �> > > Metron.
>>>> �> > >
>>>> �> > > It would be great if we could create a separate repo for the
>>>> �> plugin.
>>>> �> > > That
>>>> �> > > solves the challenge of using the packaging mechanism.
>>>> �> > >
>>>> �> > > We do need to reconcile what is in bro/bro-plugins and what is
>>>> �in
>>>> �> > > Metron.
>>>> �> > > There are some enhancements that I and others have made that
>>>> �never
>>>> �> > > made it
>>>> �> > > back into Metron. They never made it back, because the original
>>>> �> plan
>>>> �> > > was
>>>> �> > > just to switch to using the plugin from bro/bro-plugins before
>>>> �the
>>>> �> > > idea of
>>>> �> > > a packaging mechanism hit Bro. Reconciling should be fairly
>>>> �easy to
>>>> �> > > see by
>>>> �> > > just doing a diff.
>>>> �> > >
>>>> �> > > It would be great if others want to take on any of that work. I
>>>> �> would
>>>> �> > > be
>>>> �> > > glad to offer any support that you need. Thanks, Jon!
>>>> �> > >
>>>> �> > >
>>>> �> > >
>>>> �> > >
>>>> �> > >
>>>> �> > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>>>> �> zeolla@gmail.com
>>>> �> > >
>>>> �> > > wrote:
>>>> �> > >
>>>> �> > > > Ok, great.
>>>> �> > > >
>>>> �> > > > I agree, I definitely want to hear from Nick on the topic. My
>>>> �> team
>>>> �> > is
>>>> �> > > > currently looking into enhancing the plugin as well to
>>>> �> potentially
>>>> �> > > allow
>>>> �> > > > sending to multiple clusters, investigating some issues we
>>>> �see
>>>> �> when
>>>> �> > > our
>>>> �> > > bro
>>>> �> > > > cluster is under load, turn it into a package, etc.
>>>> �> > > >
>>>> �> > > > The work you just did was on our to do list as well so I'm
>>>> �very
>>>> �> > > excited
>>>> �> > > to
>>>> �> > > > see it come through.
>>>> �> > > >
>>>> �> > > > Jon
>>>> �> > > >
>>>> �> > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>>>> �cestella@gmail.com>
>>>> �> > > wrote:
>>>> �> > > >
>>>> �> > > > I *think* it's possible. People do ask for mirrors of
>>>> �directories
>>>> �> > > from
>>>> �> > > > time to time (see
>>>> �> https://issues.apache.org/jira/browse/INFRA-7060
>>>> �> > ).
>>>> �> > > If
>>>> �> > > > we
>>>> �> > > > think this is a good idea, we can pose it to INFRA as a
>>>> �request.
>>>> �> > I'd
>>>> �> > > love
>>>> �> > > > to see us be able to use the bro packaging infrastructure
>>>> �and get
>>>> �> > > more
>>>> �> > > > visibility for the plugin.
>>>> �> > > >
>>>> �> > > > I'd be particularly interested in Nick's opinion on this,
>>>> �though.
>>>> �> > > >
>>>> �> > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
>>>> �> > zeolla@gmail.com
>>>> �> > > >
>>>> �> > > > wrote:
>>>> �> > > >
>>>> �> > > > > You can version packages -
>>>> �> > > > > http://bro-package-manager.readthedocs.io/en/stable/
>>>> �> > > > package.html#package-
>>>> �> > > > > versioning
>>>> �> > > > >
>>>> �> > > > > I agree that having a separate repo provided by Apache
>>>> �would be
>>>> �> > > optimal,
>>>> �> > > > I
>>>> �> > > > > just don't know the process for that or if it was even
>>>> �> reasonable
>>>> �> > > to
>>>> �> > > > > suggest.
>>>> �> > > > >
>>>> �> > > > > Jon
>>>> �> > > > >
>>>> �> > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
>>>> �> cestella@gmail.com>
>>>> �> > > wrote:
>>>> �> > > > >
>>>> �> > > > > > Looking at the bro packages, it appears that bro is
>>>> �expecting
>>>> �> > > things
>>>> �> > > to
>>>> �> > > > > be
>>>> �> > > > > > its own git repository. I wonder if we could either
>>>> �request
>>>> �> > INFRA
>>>> �> > > > > provide
>>>> �> > > > > > another repo for the bro-kafka plugin and integrate it
>>>> �into
>>>> �> > > metron as
>>>> �> > > a
>>>> �> > > > > git
>>>> �> > > > > > submodule *or* if we could request INFRA to create a
>>>> �github
>>>> �> > > mirror of
>>>> �> > > > the
>>>> �> > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
>>>> �how
>>>> �> > > viable
>>>> �> > > > > either
>>>> �> > > > > > of those options are, frankly.
>>>> �> > > > > >
>>>> �> > > > > > One thing that I didn't see is how do you specify a
>>>> �> particular
>>>> �> > > release
>>>> �> > > > of
>>>> �> > > > > > the plugin that you want to install? For us, we'd want to
>>>> �> > > release the
>>>> �> > > > > > plugin along with the product. I didn't quite see how
>>>> �you'd
>>>> �> > push
>>>> �> > > > > releases
>>>> �> > > > > > for bro plugins.
>>>> �> > > > > >
>>>> �> > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
>>>> �> > > cestella@gmail.com>
>>>> �> > > > > wrote:
>>>> �> > > > > >
>>>> �> > > > > > > So, I do agree with the concern. Is there a way to
>>>> �host the
>>>> �> > > package
>>>> �> > > > > > > within Metron? I definitely would like to see the
>>>> �> > > modifications at
>>>> �> > > > > > > https://github.com/bro/bro-plu
>>>> �gins/commit/b9f1f35415cb0db
>>>> �> > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
>>>> �I'd
>>>> �> > > love for
>>>> �> > > > us
>>>> �> > > > > to
>>>> �> > > > > > > host the plugin.
>>>> �> > > > > > >
>>>> �> > > > > > > Thoughts?
>>>> �> > > > > > >
>>>> �> > > > > > >
>>>> �> > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
>>>> �> > > zeolla@gmail.com>
>>>> �> > >
>>>> �> > > > > > > wrote:
>>>> �> > > > > > >
>>>> �> > > > > > >> Today I was taking a look at METRON-812
>>>> �> > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
>>>> �which
>>>> �> > > made me
>>>> �> > > > > > recall
>>>> �> > > > > > >> some conversations from a while back regarding where
>>>> �the
>>>> �> bro
>>>> �> > > kafka
>>>> �> > > > > > plugin
>>>> �> > > > > > >> should ultimately live, and how to update it.
>>>> �> > > > > > >>
>>>> �> > > > > > >> Back in METRON-348 <https://issues.apache.org/
>>>> �> > > > jira/browse/METRON-348>
>>>> �> > > > > I
>>>> �> > > > > > >> brought up the fact that some important changes
>>>> �> > > > > > >> <https://github.com/bro/bro-plugins/commit/
>>>> �> > b9f1f35415cb0db06
>>>> �> > > > > > >> 5348da0a5043a8353b4a0a8>
>>>> �> > > > > > >> were made to the externally hosted version of the
>>>> �kafka
>>>> �> > > plugin,
>>>> �> > > and
>>>> �> > > > > were
>>>> �> > > > > > >> never introduced to Metron's hosted version (i.e. the
>>>> �one
>>>> �> we
>>>> �> > > use
>>>> �> > > > > > >> <https://github.com/apache/inc
>>>> �ubator-metron/blob/master/
>>>> �> > metr
>>>> �> > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>>>> �> > > > > > >> in vagrant when bro is installed). The conversation
>>>> �went
>>>> �> > down
>>>> �> > > the
>>>> �> > > > > route
>>>> �> > > > > > >> of
>>>> �> > > > > > >> discussing whether or not the bro kafka plugin code
>>>> �should
>>>> �> > > continue
>>>> �> > > > to
>>>> �> > > > > > >> live
>>>> �> > > > > > >> in Metron in the first place. Now, with METRON-812, I
>>>> �see
>>>> �> us
>>>> �> > > > further
>>>> �> > > > > > >> muddying the waters of where to go for the right
>>>> �plugin,
>>>> �> as
>>>> �> > > our
>>>> �> > > > > version
>>>> �> > > > > > is
>>>> �> > > > > > >> still missing the public changes but adds some very
>>>> �> > important
>>>> �> > > new
>>>> �> > > > > > >> functionality.
>>>> �> > > > > > >>
>>>> �> > > > > > >> I'd like to bring up the idea of using bro's packages
>>>> �> > > > > > >> <https://github.com/bro/packages> framework,
>>>> �released in
>>>> �> > > late 2016
>>>> �> > > > > > >> <
>>>> �> > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>>>> �> > > > > > >> (additional
>>>> �> > > > > > >> documentation here <
>>>> �> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>>>> �> > > > > > >> >),
>>>> �> > > > > > >> as a potential place for this to be hosted/referenced.
>>>> �> This
>>>> �> > > is a
>>>> �> > > > > simple
>>>> �> > > > > > >> and supported method (funded by Mozilla
>>>> �> > > > > > >> <https://blog.mozilla.org/blog
>>>> �/2015/12/10/mozilla-open-
>>>> �> > sourc
>>>> �> > > > > > >> e-support-first-awards-made/>)
>>>> �> > > > > > >> to install and uninstall bro scripts, plugins, etc.,
>>>> �and
>>>> �> it
>>>> �> > > also
>>>> �> > > > > allows
>>>> �> > > > > > us
>>>> �> > > > > > >> to continue to have enough control over updates to the
>>>> �> > plugin
>>>> �> > > so
>>>> �> > > > that
>>>> �> > > > > it
>>>> �> > > > > > >> will not slow down Metron development by having it as
>>>> �a
>>>> �> > > dependency
>>>> �> > > > > > >> (resolving both of Casey's concerns noted here
>>>> �> > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>>> �> > focusedCom
>>>> �> > > > > > >> mentId=15391865&page=com.atlas
>>>> �sian.jira.plugin.system.
>>>> �> > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
>>>> �> > > > > > >> and I think this solution is supported by Nick's
>>>> �comments
>>>> �> > here
>>>> �> > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>>> �> > focusedCom
>>>> �> > > > > > >> mentId=15391872&page=com.atlas
>>>> �sian.jira.plugin.system.
>>>> �> > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>>>> �> > > > > > >> as
>>>> �> > > > > > >> well).
>>>> �> > > > > > >>
>>>> �> > > > > > >> The only thing I'm not sure about is where to host the
>>>> �> > plugin
>>>> �> > > itself
>>>> �> > > > -
>>>> �> > > > > > my
>>>> �> > > > > > >> first thought would be Nick's github <
>>>> �> > > https://github.com/nickwallen
>>>> �> > > > >,
>>>> �> > > > > > as
>>>> �> > > > > > >> he
>>>> �> > > > > > >> really kicked off this effort, but maybe we can think
>>>> �of
>>>> �> > > something
>>>> �> > > > > > better.
>>>> �> > > > > > >>
>>>> �> > > > > > >> Is this approach of interest to anybody? It is
>>>> �extremely
>>>> �> > > simple to
>>>> �> > > > > put
>>>> �> > > > > > >> together - I was able to throw one together
>>>> �> > > > > > >> <https://github.com/bro/packages/blob/master/
>>>> �> > > > jonzeolla/bro-pkg.index>
>>>> �> > > > > > and
>>>> �> > > > > > >> get it working with a fresh bro 2.5 install when
>>>> �attending
>>>> �> > the
>>>> �> > > > brocon
>>>> �> > > > > > talk
>>>> �> > > > > > >> <
>>>> �> https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>>>> �> > > > > > >> bro-packagemanager>
>>>> �> > > > > > >> (recording <https://www.youtube.com/watch
>>>> �?v=9RFfPJeGkcE>,
>>>> �> > > slides
>>>> �> > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>>>> �> that
>>>> �> > > > introduced
>>>> �> > > > > > >> this
>>>> �> > > > > > >> to me in the first place.
>>>> �> > > > > > >>
>>>> �> > > > > > >> Jon
>>>> �> > > > > > >> --
>>>> �> > > > > > >>
>>>> �> > > > > > >> Jon
>>>> �> > > > > > >>
>>>> �> > > > > > >
>>>> �> > > > > > >
>>>> �> > > > > >
>>>> �> > > > > --
>>>> �> > > > >
>>>> �> > > > > Jon
>>>> �> > > > >
>>>> �> > > >
>>>> �> > > > --
>>>> �> > > >
>>>> �> > > > Jon
>>>> �> > > >
>>>> �> > >
>>>> �> > > --
>>>> �> > >
>>>> �> > > Jon
>>>> �> > >
>>>> �> > >
>>>> �> > >
>>>> �> > > --
>>>> �> > >
>>>> �> > > Jon
>>>> �> > >
>>>> �> > > --
>>>> �> > >
>>>> �> > > Jon
>>>> �> > >
>>>> �> > --
>>>> �> >
>>>> �> > Jon
>>>> �> >
>>>> �>
>>>> �--
>>>>
>>>> �Jon
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] The bro kafka plugin
Posted by Nick Allen <ni...@nickallen.org>.
Ok, I found that CouchDB and Nifi, among others, use multiple repos. A
fairly obvious pattern seems to emerge. :)
git://git.apache.org/couchdb-fabric.git
git://git.apache.org/couchdb-couch-replicator.git
git://git.apache.org/couchdb-chttpd.git
git://git.apache.org/nifi
git://git.apache.org/nifi-site.git
So what shall we call the new repo? Using
"incubator-metron-bro-plugin-kafka" is the obvious choice, but seems
excessively long. Totally open to better name for the plugin itself.
On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <ni...@nickallen.org> wrote:
> Does anyone know any other Apache projects that are using multiple repos?
> I'd like to see what they've done just so we don't break convention.
>
>
>
>
>
>
> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
>
>> Yes, I will open an INFRA ticket. Just give me a little time to research
>> what we need.
>>
>> On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>>> Okay great, thanks. Would you mind throwing in an INFRA ticket for the
>>> new
>>> repo? I can take it all from there.
>>>
>>> Does anybody know if we have ASF resources to help answer the above legal
>>> question?
>>>
>>> Jon
>>>
>>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
>>>
>>> > (1) I am not sure if licensing is a problem here.
>>> >
>>> > (2) I am OK with whatever we need to get this effort done and under
>>> ASF.
>>> >
>>> >
>>> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com>
>>> wrote:
>>> >
>>> > > I'm working on this
>>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
>>> > > preparation for the new repo and migration to a package. It looks
>>> like
>>> > in
>>> > > bro-plugins COPYING
>>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>>> > attributes
>>> > > to Nick, but in our version COPYING
>>> > > <https://github.com/apache/incubator-metron/tree/master/
>>> > > metron-sensors/bro-plugin-kafka/COPYING>
>>> > > points to the Apache License. Same with MAINTAINER (this
>>> > > <https://github.com/apache/incubator-metron/blob/master/
>>> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
>>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>>> INER
>>> > > >).
>>> > > I assume when we package this up and host it in Apache we need to
>>> give it
>>> > > the Apache license, and point to Metron for MAINTAINER. My questions
>>> > are:
>>> > >
>>> > > 1. Is there any legal/licensing concern here? I am taking changes
>>> from
>>> > > the bro-plugins version and pulling it into the Apache-hosted code.
>>> > IANAL
>>> > > 2. Nick - are you OK with these changes?
>>> > >
>>> > > Jon
>>> > >
>>> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
>>> > wrote:
>>> > >
>>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like
>>> > > > <https://www.apache.org/dev/infra-contact> committers aren't
>>> supposed
>>> > > to.
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <zeolla@gmail.com
>>> >
>>> > > wrote:
>>> > > >
>>> > > > I would be happy to try it again but I attempted to do that before
>>> with
>>> > > > bro packages and it failed to be able to handle it. I also tried
>>> using
>>> > > > branches of a repo with bro but that similarly failed (and was a
>>> pretty
>>> > > bad
>>> > > > idea to start with).
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
>>> > > >
>>> > > > We should be able to request just one alternate repo from INFRA,
>>> and
>>> > put
>>> > > a
>>> > > > top hierarchical level in it that doesn’t include a maven pom. As
>>> far
>>> > as
>>> > > > maven and clients are concerned, it
>>> > > >
>>> > > > just increases by 1 the path length to the root of the repo.
>>> > > >
>>> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>>> > > >
>>> > > > Once we agree on a repo location to host this, I would be
>>> happy to
>>> > > put
>>> > > > together the package and update our environments to use
>>> bro-pkg to
>>> > > > install
>>> > > > the plugin. I have created METRON-813
>>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track
>>> this
>>> > and
>>> > > > changed METRON-348 <
>>> > https://issues.apache.org/jira/browse/METRON-348
>>> > > >
>>> > > > to be
>>> > > > a sub-task.
>>> > > >
>>> > > > Otto - the bro packages model doesn't allow colocation with
>>> > anything
>>> > > > else.
>>> > > > That said, if we have two similar situations, and given the
>>> INFRA
>>> > > > example
>>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
>>> linked to
>>> > > > before
>>> > > > was requesting 9 repos, perhaps we just request two repos.
>>> Would
>>> > > > someone
>>> > > > else mind putting that request in?
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>>> > > ottobackwards@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > Could we create a separate repo for more than on thing? like
>>> put …
>>> > > um
>>> > > > let’s say
>>> > > > a maven plugin and the bro plugin?
>>> > > >
>>> > > >
>>> > > >
>>> > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
>>> > > wrote:
>>> > > >
>>> > > > I agree with everything that I've read.
>>> > > >
>>> > > > One of the guys from Bro had contacted me a while back,
>>> letting me
>>> > > know
>>> > > > that the packaging mechanism in Bro was ready for public
>>> > > consumption. I
>>> > > > just have not had cycles to do anything with it yet. They are
>>> not
>>> > > > wanting
>>> > > > to host any of the plugins.
>>> > > >
>>> > > > I thought the package mechanism requires that a package live
>>> within
>>> > > > its own
>>> > > > repo (which Casey confirmed). This put me in a bind on how to
>>> > tackle
>>> > > > this. I don't want to personally host the plugin in my own
>>> Github
>>> > > > repo. I
>>> > > > would prefer that we host it in a community repo; either Bro or
>>> > > Metron.
>>> > > > Since Bro is moving away from hosting their own plugins, that
>>> > leaves
>>> > > > Metron.
>>> > > >
>>> > > > It would be great if we could create a separate repo for the
>>> > plugin.
>>> > > > That
>>> > > > solves the challenge of using the packaging mechanism.
>>> > > >
>>> > > > We do need to reconcile what is in bro/bro-plugins and what is
>>> in
>>> > > > Metron.
>>> > > > There are some enhancements that I and others have made that
>>> never
>>> > > > made it
>>> > > > back into Metron. They never made it back, because the original
>>> > plan
>>> > > > was
>>> > > > just to switch to using the plugin from bro/bro-plugins before
>>> the
>>> > > > idea of
>>> > > > a packaging mechanism hit Bro. Reconciling should be fairly
>>> easy to
>>> > > > see by
>>> > > > just doing a diff.
>>> > > >
>>> > > > It would be great if others want to take on any of that work. I
>>> > would
>>> > > > be
>>> > > > glad to offer any support that you need. Thanks, Jon!
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>>> > zeolla@gmail.com
>>> > > >
>>> > > > wrote:
>>> > > >
>>> > > > > Ok, great.
>>> > > > >
>>> > > > > I agree, I definitely want to hear from Nick on the topic. My
>>> > team
>>> > > is
>>> > > > > currently looking into enhancing the plugin as well to
>>> > potentially
>>> > > > allow
>>> > > > > sending to multiple clusters, investigating some issues we
>>> see
>>> > when
>>> > > > our
>>> > > > bro
>>> > > > > cluster is under load, turn it into a package, etc.
>>> > > > >
>>> > > > > The work you just did was on our to do list as well so I'm
>>> very
>>> > > > excited
>>> > > > to
>>> > > > > see it come through.
>>> > > > >
>>> > > > > Jon
>>> > > > >
>>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>>> cestella@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > I *think* it's possible. People do ask for mirrors of
>>> directories
>>> > > > from
>>> > > > > time to time (see
>>> > https://issues.apache.org/jira/browse/INFRA-7060
>>> > > ).
>>> > > > If
>>> > > > > we
>>> > > > > think this is a good idea, we can pose it to INFRA as a
>>> request.
>>> > > I'd
>>> > > > love
>>> > > > > to see us be able to use the bro packaging infrastructure
>>> and get
>>> > > > more
>>> > > > > visibility for the plugin.
>>> > > > >
>>> > > > > I'd be particularly interested in Nick's opinion on this,
>>> though.
>>> > > > >
>>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
>>> > > zeolla@gmail.com
>>> > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > You can version packages -
>>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>>> > > > > package.html#package-
>>> > > > > > versioning
>>> > > > > >
>>> > > > > > I agree that having a separate repo provided by Apache
>>> would be
>>> > > > optimal,
>>> > > > > I
>>> > > > > > just don't know the process for that or if it was even
>>> > reasonable
>>> > > > to
>>> > > > > > suggest.
>>> > > > > >
>>> > > > > > Jon
>>> > > > > >
>>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
>>> > cestella@gmail.com>
>>> > > > wrote:
>>> > > > > >
>>> > > > > > > Looking at the bro packages, it appears that bro is
>>> expecting
>>> > > > things
>>> > > > to
>>> > > > > > be
>>> > > > > > > its own git repository. I wonder if we could either
>>> request
>>> > > INFRA
>>> > > > > > provide
>>> > > > > > > another repo for the bro-kafka plugin and integrate it
>>> into
>>> > > > metron as
>>> > > > a
>>> > > > > > git
>>> > > > > > > submodule *or* if we could request INFRA to create a
>>> github
>>> > > > mirror of
>>> > > > > the
>>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
>>> how
>>> > > > viable
>>> > > > > > either
>>> > > > > > > of those options are, frankly.
>>> > > > > > >
>>> > > > > > > One thing that I didn't see is how do you specify a
>>> > particular
>>> > > > release
>>> > > > > of
>>> > > > > > > the plugin that you want to install? For us, we'd want to
>>> > > > release the
>>> > > > > > > plugin along with the product. I didn't quite see how
>>> you'd
>>> > > push
>>> > > > > > releases
>>> > > > > > > for bro plugins.
>>> > > > > > >
>>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
>>> > > > cestella@gmail.com>
>>> > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > So, I do agree with the concern. Is there a way to
>>> host the
>>> > > > package
>>> > > > > > > > within Metron? I definitely would like to see the
>>> > > > modifications at
>>> > > > > > > > https://github.com/bro/bro-plu
>>> gins/commit/b9f1f35415cb0db
>>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
>>> I'd
>>> > > > love for
>>> > > > > us
>>> > > > > > to
>>> > > > > > > > host the plugin.
>>> > > > > > > >
>>> > > > > > > > Thoughts?
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
>>> > > > zeolla@gmail.com>
>>> > > >
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > >> Today I was taking a look at METRON-812
>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
>>> which
>>> > > > made me
>>> > > > > > > recall
>>> > > > > > > >> some conversations from a while back regarding where
>>> the
>>> > bro
>>> > > > kafka
>>> > > > > > > plugin
>>> > > > > > > >> should ultimately live, and how to update it.
>>> > > > > > > >>
>>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
>>> > > > > jira/browse/METRON-348>
>>> > > > > > I
>>> > > > > > > >> brought up the fact that some important changes
>>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
>>> > > b9f1f35415cb0db06
>>> > > > > > > >> 5348da0a5043a8353b4a0a8>
>>> > > > > > > >> were made to the externally hosted version of the
>>> kafka
>>> > > > plugin,
>>> > > > and
>>> > > > > > were
>>> > > > > > > >> never introduced to Metron's hosted version (i.e. the
>>> one
>>> > we
>>> > > > use
>>> > > > > > > >> <https://github.com/apache/inc
>>> ubator-metron/blob/master/
>>> > > metr
>>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>>> > > > > > > >> in vagrant when bro is installed). The conversation
>>> went
>>> > > down
>>> > > > the
>>> > > > > > route
>>> > > > > > > >> of
>>> > > > > > > >> discussing whether or not the bro kafka plugin code
>>> should
>>> > > > continue
>>> > > > > to
>>> > > > > > > >> live
>>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I
>>> see
>>> > us
>>> > > > > further
>>> > > > > > > >> muddying the waters of where to go for the right
>>> plugin,
>>> > as
>>> > > > our
>>> > > > > > version
>>> > > > > > > is
>>> > > > > > > >> still missing the public changes but adds some very
>>> > > important
>>> > > > new
>>> > > > > > > >> functionality.
>>> > > > > > > >>
>>> > > > > > > >> I'd like to bring up the idea of using bro's packages
>>> > > > > > > >> <https://github.com/bro/packages> framework,
>>> released in
>>> > > > late 2016
>>> > > > > > > >> <
>>> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>>> > > > > > > >> (additional
>>> > > > > > > >> documentation here <
>>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>>> > > > > > > >> >),
>>> > > > > > > >> as a potential place for this to be hosted/referenced.
>>> > This
>>> > > > is a
>>> > > > > > simple
>>> > > > > > > >> and supported method (funded by Mozilla
>>> > > > > > > >> <https://blog.mozilla.org/blog
>>> /2015/12/10/mozilla-open-
>>> > > sourc
>>> > > > > > > >> e-support-first-awards-made/>)
>>> > > > > > > >> to install and uninstall bro scripts, plugins, etc.,
>>> and
>>> > it
>>> > > > also
>>> > > > > > allows
>>> > > > > > > us
>>> > > > > > > >> to continue to have enough control over updates to the
>>> > > plugin
>>> > > > so
>>> > > > > that
>>> > > > > > it
>>> > > > > > > >> will not slow down Metron development by having it as
>>> a
>>> > > > dependency
>>> > > > > > > >> (resolving both of Casey's concerns noted here
>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>> > > focusedCom
>>> > > > > > > >> mentId=15391865&page=com.atlas
>>> sian.jira.plugin.system.
>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
>>> > > > > > > >> and I think this solution is supported by Nick's
>>> comments
>>> > > here
>>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>> > > focusedCom
>>> > > > > > > >> mentId=15391872&page=com.atlas
>>> sian.jira.plugin.system.
>>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>>> > > > > > > >> as
>>> > > > > > > >> well).
>>> > > > > > > >>
>>> > > > > > > >> The only thing I'm not sure about is where to host the
>>> > > plugin
>>> > > > itself
>>> > > > > -
>>> > > > > > > my
>>> > > > > > > >> first thought would be Nick's github <
>>> > > > https://github.com/nickwallen
>>> > > > > >,
>>> > > > > > > as
>>> > > > > > > >> he
>>> > > > > > > >> really kicked off this effort, but maybe we can think
>>> of
>>> > > > something
>>> > > > > > > better.
>>> > > > > > > >>
>>> > > > > > > >> Is this approach of interest to anybody? It is
>>> extremely
>>> > > > simple to
>>> > > > > > put
>>> > > > > > > >> together - I was able to throw one together
>>> > > > > > > >> <https://github.com/bro/packages/blob/master/
>>> > > > > jonzeolla/bro-pkg.index>
>>> > > > > > > and
>>> > > > > > > >> get it working with a fresh bro 2.5 install when
>>> attending
>>> > > the
>>> > > > > brocon
>>> > > > > > > talk
>>> > > > > > > >> <
>>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>>> > > > > > > >> bro-packagemanager>
>>> > > > > > > >> (recording <https://www.youtube.com/watch
>>> ?v=9RFfPJeGkcE>,
>>> > > > slides
>>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>>> > that
>>> > > > > introduced
>>> > > > > > > >> this
>>> > > > > > > >> to me in the first place.
>>> > > > > > > >>
>>> > > > > > > >> Jon
>>> > > > > > > >> --
>>> > > > > > > >>
>>> > > > > > > >> Jon
>>> > > > > > > >>
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > --
>>> > > > > >
>>> > > > > > Jon
>>> > > > > >
>>> > > > >
>>> > > > > --
>>> > > > >
>>> > > > > Jon
>>> > > > >
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > --
>>> > >
>>> > > Jon
>>> > >
>>> >
>>> --
>>>
>>> Jon
>>>
>>
>>
>
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Okay, I have discussed this on
<https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E>
numerous
<https://lists.apache.org/thread.html/1094762a5cf3d92d08ebda416863358fc51d2f96d675f3fc69cefb15@%3Cgeneral.incubator.apache.org%3E>
Apache
<https://lists.apache.org/thread.html/c92acd125dae05f0537d4505e0254dfa6382ca9f40edba7d2f4c6224@%3Cdev.metron.apache.org%3E>
lists
<https://lists.apache.org/thread.html/0b669b7460048076b01f87158a1895257223920d8712228c1135bd19@%3Clegal-discuss.apache.org%3E>
and it looks like we just need to have the original contributor (Nick)
submit the relevant change
<https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a8353b4a0a8>
to our code base
<https://github.com/JonZeolla/incubator-metron/tree/METRON-821/metron-sensors/bro-plugin-kafka>
via METRON-348 <https://issues.apache.org/jira/browse/METRON-348>. No
LICENSE modifications will be needed with this approach, afaict.
After that, I will happily handle everything else, including:
1. A fix for a currently unpatched thread safety issue causing segfaults (
METRON-858 <https://issues.apache.org/jira/browse/METRON-858>)
2. Migration of the kafka plugin to a bro package (METRON-813
<https://issues.apache.org/jira/browse/METRON-813>)
- Update of our Ansible playbooks to leverage the package instead of the
local code
- Removal of bro-plugin-kafka
<https://github.com/JonZeolla/incubator-metron/tree/METRON-348/metron-sensors/bro-plugin-kafka>
from the incubator-metron repo
Jon
On Wed, Apr 5, 2017 at 6:01 PM Nick Allen <ni...@nickallen.org> wrote:
> Created the INFRA request.
>
> https://issues.apache.org/jira/browse/INFRA-13828
>
>
> On Wed, Apr 5, 2017 at 5:00 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > Mentor discussion sent
> > <https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5
> > 604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E>
> >
> > Jon
> >
> > On Wed, Apr 5, 2017 at 4:34 PM Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> > > incubator-metron-bro-plugin-kafka is fine with me, in the hopes that
> > when
> > > we graduate it becomes metron-bro-plugin-kafka. We could also remove
> > > 'plugin' and make it incubator-metron-bro-kafka.
> > >
> > > Also, I will submit a [MENTORS] discussion.
> > >
> > > Jon
> > >
> > > On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <ma...@apache.org> wrote:
> > >
> > > Browsing https://git.apache.org/ shows lots of examples. A few are
> > quite
> > > prolific.
> > >
> > >
> > > On 4/5/17, 1:00 PM, "Nick Allen" <ni...@nickallen.org> wrote:
> > >
> > > Does anyone know any other Apache projects that are using multiple
> > > repos?
> > > I'd like to see what they've done just so we don't break
> convention.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org>
> > wrote:
> > >
> > > > Yes, I will open an INFRA ticket. Just give me a little time to
> > > research
> > > > what we need.
> > > >
> > > > On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <
> zeolla@gmail.com
> > >
> > > wrote:
> > > >
> > > >> Okay great, thanks. Would you mind throwing in an INFRA ticket
> > for
> > > the
> > > >> new
> > > >> repo? I can take it all from there.
> > > >>
> > > >> Does anybody know if we have ASF resources to help answer the
> > above
> > > legal
> > > >> question?
> > > >>
> > > >> Jon
> > > >>
> > > >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org>
> > > wrote:
> > > >>
> > > >> > (1) I am not sure if licensing is a problem here.
> > > >> >
> > > >> > (2) I am OK with whatever we need to get this effort done and
> > > under ASF.
> > > >> >
> > > >> >
> > > >> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > I'm working on this
> > > >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-
> > 348>
> > > in
> > > >> > > preparation for the new repo and migration to a package. It
> > > looks
> > > >> like
> > > >> > in
> > > >> > > bro-plugins COPYING
> > > >> > > <
> https://github.com/bro/bro-plugins/blob/master/kafka/COPYING
> > >
> > > >> > attributes
> > > >> > > to Nick, but in our version COPYING
> > > >> > > <https://github.com/apache/incubator-metron/tree/master/
> > > >> > > metron-sensors/bro-plugin-kafka/COPYING>
> > > >> > > points to the Apache License. Same with MAINTAINER (this
> > > >> > > <https://github.com/apache/incubator-metron/blob/master/
> > > >> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> > > >> > > vs this <
> > > https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
> > > >> INER
> > > >> > > >).
> > > >> > > I assume when we package this up and host it in Apache we
> need
> > > to
> > > >> give it
> > > >> > > the Apache license, and point to Metron for MAINTAINER. My
> > > questions
> > > >> > are:
> > > >> > >
> > > >> > > 1. Is there any legal/licensing concern here? I am taking
> > > changes
> > > >> from
> > > >> > > the bro-plugins version and pulling it into the
> Apache-hosted
> > > code.
> > > >> > IANAL
> > > >> > > 2. Nick - are you OK with these changes?
> > > >> > >
> > > >> > > Jon
> > > >> > >
> > > >> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> > > zeolla@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > Can someone on the PMC submit a ticket to INFRA? It looks
> > > like
> > > >> > > > <https://www.apache.org/dev/infra-contact> committers
> > aren't
> > > >> supposed
> > > >> > > to.
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> > > zeolla@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > I would be happy to try it again but I attempted to do
> that
> > > before
> > > >> with
> > > >> > > > bro packages and it failed to be able to handle it. I
> also
> > > tried
> > > >> using
> > > >> > > > branches of a repo with bro but that similarly failed (and
> > > was a
> > > >> pretty
> > > >> > > bad
> > > >> > > > idea to start with).
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <
> mattf@apache.org>
> > > wrote:
> > > >> > > >
> > > >> > > > We should be able to request just one alternate repo from
> > > INFRA, and
> > > >> > put
> > > >> > > a
> > > >> > > > top hierarchical level in it that doesn’t include a maven
> > > pom. As
> > > >> far
> > > >> > as
> > > >> > > > maven and clients are concerned, it
> > > >> > > >
> > > >> > > > just increases by 1 the path length to the root of the
> repo.
> > > >> > > >
> > > >> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <
> zeolla@gmail.com>
> > > wrote:
> > > >> > > >
> > > >> > > > Once we agree on a repo location to host this, I would
> > be
> > > happy
> > > >> to
> > > >> > > put
> > > >> > > > together the package and update our environments to
> use
> > > bro-pkg
> > > >> to
> > > >> > > > install
> > > >> > > > the plugin. I have created METRON-813
> > > >> > > > <https://issues.apache.org/jira/browse/METRON-813> to
> > > track
> > > >> this
> > > >> > and
> > > >> > > > changed METRON-348 <
> > > >> > https://issues.apache.org/jira/browse/METRON-348
> > > >> > > >
> > > >> > > > to be
> > > >> > > > a sub-task.
> > > >> > > >
> > > >> > > > Otto - the bro packages model doesn't allow colocation
> > > with
> > > >> > anything
> > > >> > > > else.
> > > >> > > > That said, if we have two similar situations, and
> given
> > > the
> > > >> INFRA
> > > >> > > > example
> > > >> > > > <https://issues.apache.org/jira/browse/INFRA-7060>
> > Casey
> > > >> linked to
> > > >> > > > before
> > > >> > > > was requesting 9 repos, perhaps we just request two
> > repos.
> > > >> Would
> > > >> > > > someone
> > > >> > > > else mind putting that request in?
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> > > >> > > ottobackwards@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > Could we create a separate repo for more than on
> thing?
> > > like
> > > >> put …
> > > >> > > um
> > > >> > > > let’s say
> > > >> > > > a maven plugin and the bro plugin?
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On March 31, 2017 at 12:30:25, Nick Allen (
> > > nick@nickallen.org)
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > I agree with everything that I've read.
> > > >> > > >
> > > >> > > > One of the guys from Bro had contacted me a while
> back,
> > > letting
> > > >> me
> > > >> > > know
> > > >> > > > that the packaging mechanism in Bro was ready for
> public
> > > >> > > consumption. I
> > > >> > > > just have not had cycles to do anything with it yet.
> > They
> > > are
> > > >> not
> > > >> > > > wanting
> > > >> > > > to host any of the plugins.
> > > >> > > >
> > > >> > > > I thought the package mechanism requires that a
> package
> > > live
> > > >> within
> > > >> > > > its own
> > > >> > > > repo (which Casey confirmed). This put me in a bind on
> > > how to
> > > >> > tackle
> > > >> > > > this. I don't want to personally host the plugin in my
> > own
> > > >> Github
> > > >> > > > repo. I
> > > >> > > > would prefer that we host it in a community repo;
> either
> > > Bro or
> > > >> > > Metron.
> > > >> > > > Since Bro is moving away from hosting their own
> plugins,
> > > that
> > > >> > leaves
> > > >> > > > Metron.
> > > >> > > >
> > > >> > > > It would be great if we could create a separate repo
> for
> > > the
> > > >> > plugin.
> > > >> > > > That
> > > >> > > > solves the challenge of using the packaging mechanism.
> > > >> > > >
> > > >> > > > We do need to reconcile what is in bro/bro-plugins and
> > > what is
> > > >> in
> > > >> > > > Metron.
> > > >> > > > There are some enhancements that I and others have
> made
> > > that
> > > >> never
> > > >> > > > made it
> > > >> > > > back into Metron. They never made it back, because the
> > > original
> > > >> > plan
> > > >> > > > was
> > > >> > > > just to switch to using the plugin from
> bro/bro-plugins
> > > before
> > > >> the
> > > >> > > > idea of
> > > >> > > > a packaging mechanism hit Bro. Reconciling should be
> > > fairly
> > > >> easy to
> > > >> > > > see by
> > > >> > > > just doing a diff.
> > > >> > > >
> > > >> > > > It would be great if others want to take on any of
> that
> > > work. I
> > > >> > would
> > > >> > > > be
> > > >> > > > glad to offer any support that you need. Thanks, Jon!
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> > > >> > zeolla@gmail.com
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Ok, great.
> > > >> > > > >
> > > >> > > > > I agree, I definitely want to hear from Nick on the
> > > topic. My
> > > >> > team
> > > >> > > is
> > > >> > > > > currently looking into enhancing the plugin as well
> to
> > > >> > potentially
> > > >> > > > allow
> > > >> > > > > sending to multiple clusters, investigating some
> > issues
> > > we see
> > > >> > when
> > > >> > > > our
> > > >> > > > bro
> > > >> > > > > cluster is under load, turn it into a package, etc.
> > > >> > > > >
> > > >> > > > > The work you just did was on our to do list as well
> so
> > > I'm
> > > >> very
> > > >> > > > excited
> > > >> > > > to
> > > >> > > > > see it come through.
> > > >> > > > >
> > > >> > > > > Jon
> > > >> > > > >
> > > >> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> > > >> cestella@gmail.com>
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > > I *think* it's possible. People do ask for mirrors
> of
> > > >> directories
> > > >> > > > from
> > > >> > > > > time to time (see
> > > >> > https://issues.apache.org/jira/browse/INFRA-7060
> > > >> > > ).
> > > >> > > > If
> > > >> > > > > we
> > > >> > > > > think this is a good idea, we can pose it to INFRA
> as
> > a
> > > >> request.
> > > >> > > I'd
> > > >> > > > love
> > > >> > > > > to see us be able to use the bro packaging
> > > infrastructure and
> > > >> get
> > > >> > > > more
> > > >> > > > > visibility for the plugin.
> > > >> > > > >
> > > >> > > > > I'd be particularly interested in Nick's opinion on
> > > this,
> > > >> though.
> > > >> > > > >
> > > >> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com
> <
> > > >> > > zeolla@gmail.com
> > > >> > > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > You can version packages -
> > > >> > > > > > http://bro-package-manager.
> > readthedocs.io/en/stable/
> > > >> > > > > package.html#package-
> > > >> > > > > > versioning
> > > >> > > > > >
> > > >> > > > > > I agree that having a separate repo provided by
> > Apache
> > > >> would be
> > > >> > > > optimal,
> > > >> > > > > I
> > > >> > > > > > just don't know the process for that or if it was
> > even
> > > >> > reasonable
> > > >> > > > to
> > > >> > > > > > suggest.
> > > >> > > > > >
> > > >> > > > > > Jon
> > > >> > > > > >
> > > >> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> > > >> > cestella@gmail.com>
> > > >> > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Looking at the bro packages, it appears that bro
> > is
> > > >> expecting
> > > >> > > > things
> > > >> > > > to
> > > >> > > > > > be
> > > >> > > > > > > its own git repository. I wonder if we could
> > either
> > > >> request
> > > >> > > INFRA
> > > >> > > > > > provide
> > > >> > > > > > > another repo for the bro-kafka plugin and
> > integrate
> > > it
> > > >> into
> > > >> > > > metron as
> > > >> > > > a
> > > >> > > > > > git
> > > >> > > > > > > submodule *or* if we could request INFRA to
> > create a
> > > >> github
> > > >> > > > mirror of
> > > >> > > > > the
> > > >> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm
> > not
> > > sure
> > > >> how
> > > >> > > > viable
> > > >> > > > > > either
> > > >> > > > > > > of those options are, frankly.
> > > >> > > > > > >
> > > >> > > > > > > One thing that I didn't see is how do you
> specify
> > a
> > > >> > particular
> > > >> > > > release
> > > >> > > > > of
> > > >> > > > > > > the plugin that you want to install? For us,
> we'd
> > > want to
> > > >> > > > release the
> > > >> > > > > > > plugin along with the product. I didn't quite
> see
> > > how
> > > >> you'd
> > > >> > > push
> > > >> > > > > > releases
> > > >> > > > > > > for bro plugins.
> > > >> > > > > > >
> > > >> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> > > >> > > > cestella@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > So, I do agree with the concern. Is there a
> way
> > > to host
> > > >> the
> > > >> > > > package
> > > >> > > > > > > > within Metron? I definitely would like to see
> > the
> > > >> > > > modifications at
> > > >> > > > > > > > https://github.com/bro/bro-plu
> > > >> gins/commit/b9f1f35415cb0db
> > > >> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into
> > > Metron and
> > > >> I'd
> > > >> > > > love for
> > > >> > > > > us
> > > >> > > > > > to
> > > >> > > > > > > > host the plugin.
> > > >> > > > > > > >
> > > >> > > > > > > > Thoughts?
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM,
> > Zeolla@GMail.com
> > > <
> > > >> > > > zeolla@gmail.com>
> > > >> > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > >> Today I was taking a look at METRON-812
> > > >> > > > > > > >> <
> > > https://issues.apache.org/jira/browse/METRON-812>,
> > > >> which
> > > >> > > > made me
> > > >> > > > > > > recall
> > > >> > > > > > > >> some conversations from a while back
> regarding
> > > where
> > > >> the
> > > >> > bro
> > > >> > > > kafka
> > > >> > > > > > > plugin
> > > >> > > > > > > >> should ultimately live, and how to update it.
> > > >> > > > > > > >>
> > > >> > > > > > > >> Back in METRON-348 <
> https://issues.apache.org/
> > > >> > > > > jira/browse/METRON-348>
> > > >> > > > > > I
> > > >> > > > > > > >> brought up the fact that some important
> changes
> > > >> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> > > >> > > b9f1f35415cb0db06
> > > >> > > > > > > >> 5348da0a5043a8353b4a0a8>
> > > >> > > > > > > >> were made to the externally hosted version of
> > > the kafka
> > > >> > > > plugin,
> > > >> > > > and
> > > >> > > > > > were
> > > >> > > > > > > >> never introduced to Metron's hosted version
> > > (i.e. the
> > > >> one
> > > >> > we
> > > >> > > > use
> > > >> > > > > > > >> <https://github.com/apache/inc
> > > >> ubator-metron/blob/master/
> > > >> > > metr
> > > >> > > > > > > >>
> > > on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > >> > > > > > > >> in vagrant when bro is installed). The
> > > conversation
> > > >> went
> > > >> > > down
> > > >> > > > the
> > > >> > > > > > route
> > > >> > > > > > > >> of
> > > >> > > > > > > >> discussing whether or not the bro kafka
> plugin
> > > code
> > > >> should
> > > >> > > > continue
> > > >> > > > > to
> > > >> > > > > > > >> live
> > > >> > > > > > > >> in Metron in the first place. Now, with
> > > METRON-812, I
> > > >> see
> > > >> > us
> > > >> > > > > further
> > > >> > > > > > > >> muddying the waters of where to go for the
> > right
> > > >> plugin,
> > > >> > as
> > > >> > > > our
> > > >> > > > > > version
> > > >> > > > > > > is
> > > >> > > > > > > >> still missing the public changes but adds
> some
> > > very
> > > >> > > important
> > > >> > > > new
> > > >> > > > > > > >> functionality.
> > > >> > > > > > > >>
> > > >> > > > > > > >> I'd like to bring up the idea of using bro's
> > > packages
> > > >> > > > > > > >> <https://github.com/bro/packages> framework,
> > > released
> > > >> in
> > > >> > > > late 2016
> > > >> > > > > > > >> <
> > > >> > > >
> > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > >> > > > > > > >> (additional
> > > >> > > > > > > >> documentation here <
> > > >> > > > > > >
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > > >> > > > > > > >> >),
> > > >> > > > > > > >> as a potential place for this to be
> > > hosted/referenced.
> > > >> > This
> > > >> > > > is a
> > > >> > > > > > simple
> > > >> > > > > > > >> and supported method (funded by Mozilla
> > > >> > > > > > > >> <https://blog.mozilla.org/blog
> > > >> /2015/12/10/mozilla-open-
> > > >> > > sourc
> > > >> > > > > > > >> e-support-first-awards-made/>)
> > > >> > > > > > > >> to install and uninstall bro scripts,
> plugins,
> > > etc.,
> > > >> and
> > > >> > it
> > > >> > > > also
> > > >> > > > > > allows
> > > >> > > > > > > us
> > > >> > > > > > > >> to continue to have enough control over
> updates
> > > to the
> > > >> > > plugin
> > > >> > > > so
> > > >> > > > > that
> > > >> > > > > > it
> > > >> > > > > > > >> will not slow down Metron development by
> having
> > > it as a
> > > >> > > > dependency
> > > >> > > > > > > >> (resolving both of Casey's concerns noted
> here
> > > >> > > > > > > >> <
> > > https://issues.apache.org/jira/browse/METRON-348?
> > > >> > > focusedCom
> > > >> > > > > > > >>
> > > mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > >> > > > > > > >>
> > > issuetabpanels:comment-tabpanel#comment-15391865>,
> > > >> > > > > > > >> and I think this solution is supported by
> > Nick's
> > > >> comments
> > > >> > > here
> > > >> > > > > > > >> <
> > > https://issues.apache.org/jira/browse/METRON-348?
> > > >> > > focusedCom
> > > >> > > > > > > >>
> > > mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > >> > > > > > > >> issuetabpanels:comment-
> > tabpanel#comment-15391872>
> > > >> > > > > > > >> as
> > > >> > > > > > > >> well).
> > > >> > > > > > > >>
> > > >> > > > > > > >> The only thing I'm not sure about is where to
> > > host the
> > > >> > > plugin
> > > >> > > > itself
> > > >> > > > > -
> > > >> > > > > > > my
> > > >> > > > > > > >> first thought would be Nick's github <
> > > >> > > > https://github.com/nickwallen
> > > >> > > > > >,
> > > >> > > > > > > as
> > > >> > > > > > > >> he
> > > >> > > > > > > >> really kicked off this effort, but maybe we
> can
> > > think
> > > >> of
> > > >> > > > something
> > > >> > > > > > > better.
> > > >> > > > > > > >>
> > > >> > > > > > > >> Is this approach of interest to anybody? It
> is
> > > >> extremely
> > > >> > > > simple to
> > > >> > > > > > put
> > > >> > > > > > > >> together - I was able to throw one together
> > > >> > > > > > > >> <
> https://github.com/bro/packages/blob/master/
> > > >> > > > > jonzeolla/bro-pkg.index>
> > > >> > > > > > > and
> > > >> > > > > > > >> get it working with a fresh bro 2.5 install
> > when
> > > >> attending
> > > >> > > the
> > > >> > > > > brocon
> > > >> > > > > > > talk
> > > >> > > > > > > >> <
> > > >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > >> > > > > > > >> bro-packagemanager>
> > > >> > > > > > > >> (recording <https://www.youtube.com/watch
> > > >> ?v=9RFfPJeGkcE>,
> > > >> > > > slides
> > > >> > > > > > > >> <
> > > https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> > > >> > that
> > > >> > > > > introduced
> > > >> > > > > > > >> this
> > > >> > > > > > > >> to me in the first place.
> > > >> > > > > > > >>
> > > >> > > > > > > >> Jon
> > > >> > > > > > > >> --
> > > >> > > > > > > >>
> > > >> > > > > > > >> Jon
> > > >> > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > --
> > > >> > > > > >
> > > >> > > > > > Jon
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > >
> > > >> > > > > Jon
> > > >> > > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > --
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > --
> > > >> > >
> > > >> > > Jon
> > > >> > >
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Nick Allen <ni...@nickallen.org>.
Created the INFRA request.
https://issues.apache.org/jira/browse/INFRA-13828
On Wed, Apr 5, 2017 at 5:00 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Mentor discussion sent
> <https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5
> 604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E>
>
> Jon
>
> On Wed, Apr 5, 2017 at 4:34 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > incubator-metron-bro-plugin-kafka is fine with me, in the hopes that
> when
> > we graduate it becomes metron-bro-plugin-kafka. We could also remove
> > 'plugin' and make it incubator-metron-bro-kafka.
> >
> > Also, I will submit a [MENTORS] discussion.
> >
> > Jon
> >
> > On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <ma...@apache.org> wrote:
> >
> > Browsing https://git.apache.org/ shows lots of examples. A few are
> quite
> > prolific.
> >
> >
> > On 4/5/17, 1:00 PM, "Nick Allen" <ni...@nickallen.org> wrote:
> >
> > Does anyone know any other Apache projects that are using multiple
> > repos?
> > I'd like to see what they've done just so we don't break convention.
> >
> >
> >
> >
> >
> >
> > On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org>
> wrote:
> >
> > > Yes, I will open an INFRA ticket. Just give me a little time to
> > research
> > > what we need.
> > >
> > > On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> > >
> > >> Okay great, thanks. Would you mind throwing in an INFRA ticket
> for
> > the
> > >> new
> > >> repo? I can take it all from there.
> > >>
> > >> Does anybody know if we have ASF resources to help answer the
> above
> > legal
> > >> question?
> > >>
> > >> Jon
> > >>
> > >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org>
> > wrote:
> > >>
> > >> > (1) I am not sure if licensing is a problem here.
> > >> >
> > >> > (2) I am OK with whatever we need to get this effort done and
> > under ASF.
> > >> >
> > >> >
> > >> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <
> > zeolla@gmail.com>
> > >> wrote:
> > >> >
> > >> > > I'm working on this
> > >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-
> 348>
> > in
> > >> > > preparation for the new repo and migration to a package. It
> > looks
> > >> like
> > >> > in
> > >> > > bro-plugins COPYING
> > >> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING
> >
> > >> > attributes
> > >> > > to Nick, but in our version COPYING
> > >> > > <https://github.com/apache/incubator-metron/tree/master/
> > >> > > metron-sensors/bro-plugin-kafka/COPYING>
> > >> > > points to the Apache License. Same with MAINTAINER (this
> > >> > > <https://github.com/apache/incubator-metron/blob/master/
> > >> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> > >> > > vs this <
> > https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
> > >> INER
> > >> > > >).
> > >> > > I assume when we package this up and host it in Apache we need
> > to
> > >> give it
> > >> > > the Apache license, and point to Metron for MAINTAINER. My
> > questions
> > >> > are:
> > >> > >
> > >> > > 1. Is there any legal/licensing concern here? I am taking
> > changes
> > >> from
> > >> > > the bro-plugins version and pulling it into the Apache-hosted
> > code.
> > >> > IANAL
> > >> > > 2. Nick - are you OK with these changes?
> > >> > >
> > >> > > Jon
> > >> > >
> > >> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> > zeolla@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > Can someone on the PMC submit a ticket to INFRA? It looks
> > like
> > >> > > > <https://www.apache.org/dev/infra-contact> committers
> aren't
> > >> supposed
> > >> > > to.
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> > zeolla@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > I would be happy to try it again but I attempted to do that
> > before
> > >> with
> > >> > > > bro packages and it failed to be able to handle it. I also
> > tried
> > >> using
> > >> > > > branches of a repo with bro but that similarly failed (and
> > was a
> > >> pretty
> > >> > > bad
> > >> > > > idea to start with).
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org>
> > wrote:
> > >> > > >
> > >> > > > We should be able to request just one alternate repo from
> > INFRA, and
> > >> > put
> > >> > > a
> > >> > > > top hierarchical level in it that doesn’t include a maven
> > pom. As
> > >> far
> > >> > as
> > >> > > > maven and clients are concerned, it
> > >> > > >
> > >> > > > just increases by 1 the path length to the root of the repo.
> > >> > > >
> > >> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com>
> > wrote:
> > >> > > >
> > >> > > > Once we agree on a repo location to host this, I would
> be
> > happy
> > >> to
> > >> > > put
> > >> > > > together the package and update our environments to use
> > bro-pkg
> > >> to
> > >> > > > install
> > >> > > > the plugin. I have created METRON-813
> > >> > > > <https://issues.apache.org/jira/browse/METRON-813> to
> > track
> > >> this
> > >> > and
> > >> > > > changed METRON-348 <
> > >> > https://issues.apache.org/jira/browse/METRON-348
> > >> > > >
> > >> > > > to be
> > >> > > > a sub-task.
> > >> > > >
> > >> > > > Otto - the bro packages model doesn't allow colocation
> > with
> > >> > anything
> > >> > > > else.
> > >> > > > That said, if we have two similar situations, and given
> > the
> > >> INFRA
> > >> > > > example
> > >> > > > <https://issues.apache.org/jira/browse/INFRA-7060>
> Casey
> > >> linked to
> > >> > > > before
> > >> > > > was requesting 9 repos, perhaps we just request two
> repos.
> > >> Would
> > >> > > > someone
> > >> > > > else mind putting that request in?
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> > >> > > ottobackwards@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > Could we create a separate repo for more than on thing?
> > like
> > >> put …
> > >> > > um
> > >> > > > let’s say
> > >> > > > a maven plugin and the bro plugin?
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On March 31, 2017 at 12:30:25, Nick Allen (
> > nick@nickallen.org)
> > >> > > wrote:
> > >> > > >
> > >> > > > I agree with everything that I've read.
> > >> > > >
> > >> > > > One of the guys from Bro had contacted me a while back,
> > letting
> > >> me
> > >> > > know
> > >> > > > that the packaging mechanism in Bro was ready for public
> > >> > > consumption. I
> > >> > > > just have not had cycles to do anything with it yet.
> They
> > are
> > >> not
> > >> > > > wanting
> > >> > > > to host any of the plugins.
> > >> > > >
> > >> > > > I thought the package mechanism requires that a package
> > live
> > >> within
> > >> > > > its own
> > >> > > > repo (which Casey confirmed). This put me in a bind on
> > how to
> > >> > tackle
> > >> > > > this. I don't want to personally host the plugin in my
> own
> > >> Github
> > >> > > > repo. I
> > >> > > > would prefer that we host it in a community repo; either
> > Bro or
> > >> > > Metron.
> > >> > > > Since Bro is moving away from hosting their own plugins,
> > that
> > >> > leaves
> > >> > > > Metron.
> > >> > > >
> > >> > > > It would be great if we could create a separate repo for
> > the
> > >> > plugin.
> > >> > > > That
> > >> > > > solves the challenge of using the packaging mechanism.
> > >> > > >
> > >> > > > We do need to reconcile what is in bro/bro-plugins and
> > what is
> > >> in
> > >> > > > Metron.
> > >> > > > There are some enhancements that I and others have made
> > that
> > >> never
> > >> > > > made it
> > >> > > > back into Metron. They never made it back, because the
> > original
> > >> > plan
> > >> > > > was
> > >> > > > just to switch to using the plugin from bro/bro-plugins
> > before
> > >> the
> > >> > > > idea of
> > >> > > > a packaging mechanism hit Bro. Reconciling should be
> > fairly
> > >> easy to
> > >> > > > see by
> > >> > > > just doing a diff.
> > >> > > >
> > >> > > > It would be great if others want to take on any of that
> > work. I
> > >> > would
> > >> > > > be
> > >> > > > glad to offer any support that you need. Thanks, Jon!
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> > >> > zeolla@gmail.com
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Ok, great.
> > >> > > > >
> > >> > > > > I agree, I definitely want to hear from Nick on the
> > topic. My
> > >> > team
> > >> > > is
> > >> > > > > currently looking into enhancing the plugin as well to
> > >> > potentially
> > >> > > > allow
> > >> > > > > sending to multiple clusters, investigating some
> issues
> > we see
> > >> > when
> > >> > > > our
> > >> > > > bro
> > >> > > > > cluster is under load, turn it into a package, etc.
> > >> > > > >
> > >> > > > > The work you just did was on our to do list as well so
> > I'm
> > >> very
> > >> > > > excited
> > >> > > > to
> > >> > > > > see it come through.
> > >> > > > >
> > >> > > > > Jon
> > >> > > > >
> > >> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> > >> cestella@gmail.com>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > I *think* it's possible. People do ask for mirrors of
> > >> directories
> > >> > > > from
> > >> > > > > time to time (see
> > >> > https://issues.apache.org/jira/browse/INFRA-7060
> > >> > > ).
> > >> > > > If
> > >> > > > > we
> > >> > > > > think this is a good idea, we can pose it to INFRA as
> a
> > >> request.
> > >> > > I'd
> > >> > > > love
> > >> > > > > to see us be able to use the bro packaging
> > infrastructure and
> > >> get
> > >> > > > more
> > >> > > > > visibility for the plugin.
> > >> > > > >
> > >> > > > > I'd be particularly interested in Nick's opinion on
> > this,
> > >> though.
> > >> > > > >
> > >> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> > >> > > zeolla@gmail.com
> > >> > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > You can version packages -
> > >> > > > > > http://bro-package-manager.
> readthedocs.io/en/stable/
> > >> > > > > package.html#package-
> > >> > > > > > versioning
> > >> > > > > >
> > >> > > > > > I agree that having a separate repo provided by
> Apache
> > >> would be
> > >> > > > optimal,
> > >> > > > > I
> > >> > > > > > just don't know the process for that or if it was
> even
> > >> > reasonable
> > >> > > > to
> > >> > > > > > suggest.
> > >> > > > > >
> > >> > > > > > Jon
> > >> > > > > >
> > >> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> > >> > cestella@gmail.com>
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > > > Looking at the bro packages, it appears that bro
> is
> > >> expecting
> > >> > > > things
> > >> > > > to
> > >> > > > > > be
> > >> > > > > > > its own git repository. I wonder if we could
> either
> > >> request
> > >> > > INFRA
> > >> > > > > > provide
> > >> > > > > > > another repo for the bro-kafka plugin and
> integrate
> > it
> > >> into
> > >> > > > metron as
> > >> > > > a
> > >> > > > > > git
> > >> > > > > > > submodule *or* if we could request INFRA to
> create a
> > >> github
> > >> > > > mirror of
> > >> > > > > the
> > >> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm
> not
> > sure
> > >> how
> > >> > > > viable
> > >> > > > > > either
> > >> > > > > > > of those options are, frankly.
> > >> > > > > > >
> > >> > > > > > > One thing that I didn't see is how do you specify
> a
> > >> > particular
> > >> > > > release
> > >> > > > > of
> > >> > > > > > > the plugin that you want to install? For us, we'd
> > want to
> > >> > > > release the
> > >> > > > > > > plugin along with the product. I didn't quite see
> > how
> > >> you'd
> > >> > > push
> > >> > > > > > releases
> > >> > > > > > > for bro plugins.
> > >> > > > > > >
> > >> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> > >> > > > cestella@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > So, I do agree with the concern. Is there a way
> > to host
> > >> the
> > >> > > > package
> > >> > > > > > > > within Metron? I definitely would like to see
> the
> > >> > > > modifications at
> > >> > > > > > > > https://github.com/bro/bro-plu
> > >> gins/commit/b9f1f35415cb0db
> > >> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into
> > Metron and
> > >> I'd
> > >> > > > love for
> > >> > > > > us
> > >> > > > > > to
> > >> > > > > > > > host the plugin.
> > >> > > > > > > >
> > >> > > > > > > > Thoughts?
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM,
> Zeolla@GMail.com
> > <
> > >> > > > zeolla@gmail.com>
> > >> > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > >> Today I was taking a look at METRON-812
> > >> > > > > > > >> <
> > https://issues.apache.org/jira/browse/METRON-812>,
> > >> which
> > >> > > > made me
> > >> > > > > > > recall
> > >> > > > > > > >> some conversations from a while back regarding
> > where
> > >> the
> > >> > bro
> > >> > > > kafka
> > >> > > > > > > plugin
> > >> > > > > > > >> should ultimately live, and how to update it.
> > >> > > > > > > >>
> > >> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
> > >> > > > > jira/browse/METRON-348>
> > >> > > > > > I
> > >> > > > > > > >> brought up the fact that some important changes
> > >> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> > >> > > b9f1f35415cb0db06
> > >> > > > > > > >> 5348da0a5043a8353b4a0a8>
> > >> > > > > > > >> were made to the externally hosted version of
> > the kafka
> > >> > > > plugin,
> > >> > > > and
> > >> > > > > > were
> > >> > > > > > > >> never introduced to Metron's hosted version
> > (i.e. the
> > >> one
> > >> > we
> > >> > > > use
> > >> > > > > > > >> <https://github.com/apache/inc
> > >> ubator-metron/blob/master/
> > >> > > metr
> > >> > > > > > > >>
> > on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > >> > > > > > > >> in vagrant when bro is installed). The
> > conversation
> > >> went
> > >> > > down
> > >> > > > the
> > >> > > > > > route
> > >> > > > > > > >> of
> > >> > > > > > > >> discussing whether or not the bro kafka plugin
> > code
> > >> should
> > >> > > > continue
> > >> > > > > to
> > >> > > > > > > >> live
> > >> > > > > > > >> in Metron in the first place. Now, with
> > METRON-812, I
> > >> see
> > >> > us
> > >> > > > > further
> > >> > > > > > > >> muddying the waters of where to go for the
> right
> > >> plugin,
> > >> > as
> > >> > > > our
> > >> > > > > > version
> > >> > > > > > > is
> > >> > > > > > > >> still missing the public changes but adds some
> > very
> > >> > > important
> > >> > > > new
> > >> > > > > > > >> functionality.
> > >> > > > > > > >>
> > >> > > > > > > >> I'd like to bring up the idea of using bro's
> > packages
> > >> > > > > > > >> <https://github.com/bro/packages> framework,
> > released
> > >> in
> > >> > > > late 2016
> > >> > > > > > > >> <
> > >> > > >
> > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > >> > > > > > > >> (additional
> > >> > > > > > > >> documentation here <
> > >> > > > > > >
> > http://bro-package-manager.readthedocs.io/en/stable/
> > >> > > > > > > >> >),
> > >> > > > > > > >> as a potential place for this to be
> > hosted/referenced.
> > >> > This
> > >> > > > is a
> > >> > > > > > simple
> > >> > > > > > > >> and supported method (funded by Mozilla
> > >> > > > > > > >> <https://blog.mozilla.org/blog
> > >> /2015/12/10/mozilla-open-
> > >> > > sourc
> > >> > > > > > > >> e-support-first-awards-made/>)
> > >> > > > > > > >> to install and uninstall bro scripts, plugins,
> > etc.,
> > >> and
> > >> > it
> > >> > > > also
> > >> > > > > > allows
> > >> > > > > > > us
> > >> > > > > > > >> to continue to have enough control over updates
> > to the
> > >> > > plugin
> > >> > > > so
> > >> > > > > that
> > >> > > > > > it
> > >> > > > > > > >> will not slow down Metron development by having
> > it as a
> > >> > > > dependency
> > >> > > > > > > >> (resolving both of Casey's concerns noted here
> > >> > > > > > > >> <
> > https://issues.apache.org/jira/browse/METRON-348?
> > >> > > focusedCom
> > >> > > > > > > >>
> > mentId=15391865&page=com.atlassian.jira.plugin.system.
> > >> > > > > > > >>
> > issuetabpanels:comment-tabpanel#comment-15391865>,
> > >> > > > > > > >> and I think this solution is supported by
> Nick's
> > >> comments
> > >> > > here
> > >> > > > > > > >> <
> > https://issues.apache.org/jira/browse/METRON-348?
> > >> > > focusedCom
> > >> > > > > > > >>
> > mentId=15391872&page=com.atlassian.jira.plugin.system.
> > >> > > > > > > >> issuetabpanels:comment-
> tabpanel#comment-15391872>
> > >> > > > > > > >> as
> > >> > > > > > > >> well).
> > >> > > > > > > >>
> > >> > > > > > > >> The only thing I'm not sure about is where to
> > host the
> > >> > > plugin
> > >> > > > itself
> > >> > > > > -
> > >> > > > > > > my
> > >> > > > > > > >> first thought would be Nick's github <
> > >> > > > https://github.com/nickwallen
> > >> > > > > >,
> > >> > > > > > > as
> > >> > > > > > > >> he
> > >> > > > > > > >> really kicked off this effort, but maybe we can
> > think
> > >> of
> > >> > > > something
> > >> > > > > > > better.
> > >> > > > > > > >>
> > >> > > > > > > >> Is this approach of interest to anybody? It is
> > >> extremely
> > >> > > > simple to
> > >> > > > > > put
> > >> > > > > > > >> together - I was able to throw one together
> > >> > > > > > > >> <https://github.com/bro/packages/blob/master/
> > >> > > > > jonzeolla/bro-pkg.index>
> > >> > > > > > > and
> > >> > > > > > > >> get it working with a fresh bro 2.5 install
> when
> > >> attending
> > >> > > the
> > >> > > > > brocon
> > >> > > > > > > talk
> > >> > > > > > > >> <
> > >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > >> > > > > > > >> bro-packagemanager>
> > >> > > > > > > >> (recording <https://www.youtube.com/watch
> > >> ?v=9RFfPJeGkcE>,
> > >> > > > slides
> > >> > > > > > > >> <
> > https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> > >> > that
> > >> > > > > introduced
> > >> > > > > > > >> this
> > >> > > > > > > >> to me in the first place.
> > >> > > > > > > >>
> > >> > > > > > > >> Jon
> > >> > > > > > > >> --
> > >> > > > > > > >>
> > >> > > > > > > >> Jon
> > >> > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > --
> > >> > > > > >
> > >> > > > > > Jon
> > >> > > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > >
> > >> > > > > Jon
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > > --
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > --
> > >> > >
> > >> > > Jon
> > >> > >
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> >
> >
> > --
> >
> > Jon
> >
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Mentor discussion sent
<https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E>
Jon
On Wed, Apr 5, 2017 at 4:34 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
> incubator-metron-bro-plugin-kafka is fine with me, in the hopes that when
> we graduate it becomes metron-bro-plugin-kafka. We could also remove
> 'plugin' and make it incubator-metron-bro-kafka.
>
> Also, I will submit a [MENTORS] discussion.
>
> Jon
>
> On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <ma...@apache.org> wrote:
>
> Browsing https://git.apache.org/ shows lots of examples. A few are quite
> prolific.
>
>
> On 4/5/17, 1:00 PM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> Does anyone know any other Apache projects that are using multiple
> repos?
> I'd like to see what they've done just so we don't break convention.
>
>
>
>
>
>
> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > Yes, I will open an INFRA ticket. Just give me a little time to
> research
> > what we need.
> >
> > On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> >> Okay great, thanks. Would you mind throwing in an INFRA ticket for
> the
> >> new
> >> repo? I can take it all from there.
> >>
> >> Does anybody know if we have ASF resources to help answer the above
> legal
> >> question?
> >>
> >> Jon
> >>
> >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org>
> wrote:
> >>
> >> > (1) I am not sure if licensing is a problem here.
> >> >
> >> > (2) I am OK with whatever we need to get this effort done and
> under ASF.
> >> >
> >> >
> >> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
> >> wrote:
> >> >
> >> > > I'm working on this
> >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348>
> in
> >> > > preparation for the new repo and migration to a package. It
> looks
> >> like
> >> > in
> >> > > bro-plugins COPYING
> >> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
> >> > attributes
> >> > > to Nick, but in our version COPYING
> >> > > <https://github.com/apache/incubator-metron/tree/master/
> >> > > metron-sensors/bro-plugin-kafka/COPYING>
> >> > > points to the Apache License. Same with MAINTAINER (this
> >> > > <https://github.com/apache/incubator-metron/blob/master/
> >> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> >> > > vs this <
> https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
> >> INER
> >> > > >).
> >> > > I assume when we package this up and host it in Apache we need
> to
> >> give it
> >> > > the Apache license, and point to Metron for MAINTAINER. My
> questions
> >> > are:
> >> > >
> >> > > 1. Is there any legal/licensing concern here? I am taking
> changes
> >> from
> >> > > the bro-plugins version and pulling it into the Apache-hosted
> code.
> >> > IANAL
> >> > > 2. Nick - are you OK with these changes?
> >> > >
> >> > > Jon
> >> > >
> >> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> zeolla@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Can someone on the PMC submit a ticket to INFRA? It looks
> like
> >> > > > <https://www.apache.org/dev/infra-contact> committers aren't
> >> supposed
> >> > > to.
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> zeolla@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > I would be happy to try it again but I attempted to do that
> before
> >> with
> >> > > > bro packages and it failed to be able to handle it. I also
> tried
> >> using
> >> > > > branches of a repo with bro but that similarly failed (and
> was a
> >> pretty
> >> > > bad
> >> > > > idea to start with).
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org>
> wrote:
> >> > > >
> >> > > > We should be able to request just one alternate repo from
> INFRA, and
> >> > put
> >> > > a
> >> > > > top hierarchical level in it that doesn’t include a maven
> pom. As
> >> far
> >> > as
> >> > > > maven and clients are concerned, it
> >> > > >
> >> > > > just increases by 1 the path length to the root of the repo.
> >> > > >
> >> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com>
> wrote:
> >> > > >
> >> > > > Once we agree on a repo location to host this, I would be
> happy
> >> to
> >> > > put
> >> > > > together the package and update our environments to use
> bro-pkg
> >> to
> >> > > > install
> >> > > > the plugin. I have created METRON-813
> >> > > > <https://issues.apache.org/jira/browse/METRON-813> to
> track
> >> this
> >> > and
> >> > > > changed METRON-348 <
> >> > https://issues.apache.org/jira/browse/METRON-348
> >> > > >
> >> > > > to be
> >> > > > a sub-task.
> >> > > >
> >> > > > Otto - the bro packages model doesn't allow colocation
> with
> >> > anything
> >> > > > else.
> >> > > > That said, if we have two similar situations, and given
> the
> >> INFRA
> >> > > > example
> >> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
> >> linked to
> >> > > > before
> >> > > > was requesting 9 repos, perhaps we just request two repos.
> >> Would
> >> > > > someone
> >> > > > else mind putting that request in?
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> >> > > ottobackwards@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > Could we create a separate repo for more than on thing?
> like
> >> put …
> >> > > um
> >> > > > let’s say
> >> > > > a maven plugin and the bro plugin?
> >> > > >
> >> > > >
> >> > > >
> >> > > > On March 31, 2017 at 12:30:25, Nick Allen (
> nick@nickallen.org)
> >> > > wrote:
> >> > > >
> >> > > > I agree with everything that I've read.
> >> > > >
> >> > > > One of the guys from Bro had contacted me a while back,
> letting
> >> me
> >> > > know
> >> > > > that the packaging mechanism in Bro was ready for public
> >> > > consumption. I
> >> > > > just have not had cycles to do anything with it yet. They
> are
> >> not
> >> > > > wanting
> >> > > > to host any of the plugins.
> >> > > >
> >> > > > I thought the package mechanism requires that a package
> live
> >> within
> >> > > > its own
> >> > > > repo (which Casey confirmed). This put me in a bind on
> how to
> >> > tackle
> >> > > > this. I don't want to personally host the plugin in my own
> >> Github
> >> > > > repo. I
> >> > > > would prefer that we host it in a community repo; either
> Bro or
> >> > > Metron.
> >> > > > Since Bro is moving away from hosting their own plugins,
> that
> >> > leaves
> >> > > > Metron.
> >> > > >
> >> > > > It would be great if we could create a separate repo for
> the
> >> > plugin.
> >> > > > That
> >> > > > solves the challenge of using the packaging mechanism.
> >> > > >
> >> > > > We do need to reconcile what is in bro/bro-plugins and
> what is
> >> in
> >> > > > Metron.
> >> > > > There are some enhancements that I and others have made
> that
> >> never
> >> > > > made it
> >> > > > back into Metron. They never made it back, because the
> original
> >> > plan
> >> > > > was
> >> > > > just to switch to using the plugin from bro/bro-plugins
> before
> >> the
> >> > > > idea of
> >> > > > a packaging mechanism hit Bro. Reconciling should be
> fairly
> >> easy to
> >> > > > see by
> >> > > > just doing a diff.
> >> > > >
> >> > > > It would be great if others want to take on any of that
> work. I
> >> > would
> >> > > > be
> >> > > > glad to offer any support that you need. Thanks, Jon!
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> >> > zeolla@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Ok, great.
> >> > > > >
> >> > > > > I agree, I definitely want to hear from Nick on the
> topic. My
> >> > team
> >> > > is
> >> > > > > currently looking into enhancing the plugin as well to
> >> > potentially
> >> > > > allow
> >> > > > > sending to multiple clusters, investigating some issues
> we see
> >> > when
> >> > > > our
> >> > > > bro
> >> > > > > cluster is under load, turn it into a package, etc.
> >> > > > >
> >> > > > > The work you just did was on our to do list as well so
> I'm
> >> very
> >> > > > excited
> >> > > > to
> >> > > > > see it come through.
> >> > > > >
> >> > > > > Jon
> >> > > > >
> >> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> >> cestella@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > I *think* it's possible. People do ask for mirrors of
> >> directories
> >> > > > from
> >> > > > > time to time (see
> >> > https://issues.apache.org/jira/browse/INFRA-7060
> >> > > ).
> >> > > > If
> >> > > > > we
> >> > > > > think this is a good idea, we can pose it to INFRA as a
> >> request.
> >> > > I'd
> >> > > > love
> >> > > > > to see us be able to use the bro packaging
> infrastructure and
> >> get
> >> > > > more
> >> > > > > visibility for the plugin.
> >> > > > >
> >> > > > > I'd be particularly interested in Nick's opinion on
> this,
> >> though.
> >> > > > >
> >> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> >> > > zeolla@gmail.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > You can version packages -
> >> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> >> > > > > package.html#package-
> >> > > > > > versioning
> >> > > > > >
> >> > > > > > I agree that having a separate repo provided by Apache
> >> would be
> >> > > > optimal,
> >> > > > > I
> >> > > > > > just don't know the process for that or if it was even
> >> > reasonable
> >> > > > to
> >> > > > > > suggest.
> >> > > > > >
> >> > > > > > Jon
> >> > > > > >
> >> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> >> > cestella@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Looking at the bro packages, it appears that bro is
> >> expecting
> >> > > > things
> >> > > > to
> >> > > > > > be
> >> > > > > > > its own git repository. I wonder if we could either
> >> request
> >> > > INFRA
> >> > > > > > provide
> >> > > > > > > another repo for the bro-kafka plugin and integrate
> it
> >> into
> >> > > > metron as
> >> > > > a
> >> > > > > > git
> >> > > > > > > submodule *or* if we could request INFRA to create a
> >> github
> >> > > > mirror of
> >> > > > > the
> >> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not
> sure
> >> how
> >> > > > viable
> >> > > > > > either
> >> > > > > > > of those options are, frankly.
> >> > > > > > >
> >> > > > > > > One thing that I didn't see is how do you specify a
> >> > particular
> >> > > > release
> >> > > > > of
> >> > > > > > > the plugin that you want to install? For us, we'd
> want to
> >> > > > release the
> >> > > > > > > plugin along with the product. I didn't quite see
> how
> >> you'd
> >> > > push
> >> > > > > > releases
> >> > > > > > > for bro plugins.
> >> > > > > > >
> >> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> >> > > > cestella@gmail.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > So, I do agree with the concern. Is there a way
> to host
> >> the
> >> > > > package
> >> > > > > > > > within Metron? I definitely would like to see the
> >> > > > modifications at
> >> > > > > > > > https://github.com/bro/bro-plu
> >> gins/commit/b9f1f35415cb0db
> >> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into
> Metron and
> >> I'd
> >> > > > love for
> >> > > > > us
> >> > > > > > to
> >> > > > > > > > host the plugin.
> >> > > > > > > >
> >> > > > > > > > Thoughts?
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com
> <
> >> > > > zeolla@gmail.com>
> >> > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Today I was taking a look at METRON-812
> >> > > > > > > >> <
> https://issues.apache.org/jira/browse/METRON-812>,
> >> which
> >> > > > made me
> >> > > > > > > recall
> >> > > > > > > >> some conversations from a while back regarding
> where
> >> the
> >> > bro
> >> > > > kafka
> >> > > > > > > plugin
> >> > > > > > > >> should ultimately live, and how to update it.
> >> > > > > > > >>
> >> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
> >> > > > > jira/browse/METRON-348>
> >> > > > > > I
> >> > > > > > > >> brought up the fact that some important changes
> >> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> >> > > b9f1f35415cb0db06
> >> > > > > > > >> 5348da0a5043a8353b4a0a8>
> >> > > > > > > >> were made to the externally hosted version of
> the kafka
> >> > > > plugin,
> >> > > > and
> >> > > > > > were
> >> > > > > > > >> never introduced to Metron's hosted version
> (i.e. the
> >> one
> >> > we
> >> > > > use
> >> > > > > > > >> <https://github.com/apache/inc
> >> ubator-metron/blob/master/
> >> > > metr
> >> > > > > > > >>
> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> >> > > > > > > >> in vagrant when bro is installed). The
> conversation
> >> went
> >> > > down
> >> > > > the
> >> > > > > > route
> >> > > > > > > >> of
> >> > > > > > > >> discussing whether or not the bro kafka plugin
> code
> >> should
> >> > > > continue
> >> > > > > to
> >> > > > > > > >> live
> >> > > > > > > >> in Metron in the first place. Now, with
> METRON-812, I
> >> see
> >> > us
> >> > > > > further
> >> > > > > > > >> muddying the waters of where to go for the right
> >> plugin,
> >> > as
> >> > > > our
> >> > > > > > version
> >> > > > > > > is
> >> > > > > > > >> still missing the public changes but adds some
> very
> >> > > important
> >> > > > new
> >> > > > > > > >> functionality.
> >> > > > > > > >>
> >> > > > > > > >> I'd like to bring up the idea of using bro's
> packages
> >> > > > > > > >> <https://github.com/bro/packages> framework,
> released
> >> in
> >> > > > late 2016
> >> > > > > > > >> <
> >> > > >
> http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> >> > > > > > > >> (additional
> >> > > > > > > >> documentation here <
> >> > > > > > >
> http://bro-package-manager.readthedocs.io/en/stable/
> >> > > > > > > >> >),
> >> > > > > > > >> as a potential place for this to be
> hosted/referenced.
> >> > This
> >> > > > is a
> >> > > > > > simple
> >> > > > > > > >> and supported method (funded by Mozilla
> >> > > > > > > >> <https://blog.mozilla.org/blog
> >> /2015/12/10/mozilla-open-
> >> > > sourc
> >> > > > > > > >> e-support-first-awards-made/>)
> >> > > > > > > >> to install and uninstall bro scripts, plugins,
> etc.,
> >> and
> >> > it
> >> > > > also
> >> > > > > > allows
> >> > > > > > > us
> >> > > > > > > >> to continue to have enough control over updates
> to the
> >> > > plugin
> >> > > > so
> >> > > > > that
> >> > > > > > it
> >> > > > > > > >> will not slow down Metron development by having
> it as a
> >> > > > dependency
> >> > > > > > > >> (resolving both of Casey's concerns noted here
> >> > > > > > > >> <
> https://issues.apache.org/jira/browse/METRON-348?
> >> > > focusedCom
> >> > > > > > > >>
> mentId=15391865&page=com.atlassian.jira.plugin.system.
> >> > > > > > > >>
> issuetabpanels:comment-tabpanel#comment-15391865>,
> >> > > > > > > >> and I think this solution is supported by Nick's
> >> comments
> >> > > here
> >> > > > > > > >> <
> https://issues.apache.org/jira/browse/METRON-348?
> >> > > focusedCom
> >> > > > > > > >>
> mentId=15391872&page=com.atlassian.jira.plugin.system.
> >> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> >> > > > > > > >> as
> >> > > > > > > >> well).
> >> > > > > > > >>
> >> > > > > > > >> The only thing I'm not sure about is where to
> host the
> >> > > plugin
> >> > > > itself
> >> > > > > -
> >> > > > > > > my
> >> > > > > > > >> first thought would be Nick's github <
> >> > > > https://github.com/nickwallen
> >> > > > > >,
> >> > > > > > > as
> >> > > > > > > >> he
> >> > > > > > > >> really kicked off this effort, but maybe we can
> think
> >> of
> >> > > > something
> >> > > > > > > better.
> >> > > > > > > >>
> >> > > > > > > >> Is this approach of interest to anybody? It is
> >> extremely
> >> > > > simple to
> >> > > > > > put
> >> > > > > > > >> together - I was able to throw one together
> >> > > > > > > >> <https://github.com/bro/packages/blob/master/
> >> > > > > jonzeolla/bro-pkg.index>
> >> > > > > > > and
> >> > > > > > > >> get it working with a fresh bro 2.5 install when
> >> attending
> >> > > the
> >> > > > > brocon
> >> > > > > > > talk
> >> > > > > > > >> <
> >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> >> > > > > > > >> bro-packagemanager>
> >> > > > > > > >> (recording <https://www.youtube.com/watch
> >> ?v=9RFfPJeGkcE>,
> >> > > > slides
> >> > > > > > > >> <
> https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> >> > that
> >> > > > > introduced
> >> > > > > > > >> this
> >> > > > > > > >> to me in the first place.
> >> > > > > > > >>
> >> > > > > > > >> Jon
> >> > > > > > > >> --
> >> > > > > > > >>
> >> > > > > > > >> Jon
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > --
> >> > > > > >
> >> > > > > > Jon
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > > Jon
> >> > > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > --
> >> > >
> >> > > Jon
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
>
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
incubator-metron-bro-plugin-kafka is fine with me, in the hopes that when
we graduate it becomes metron-bro-plugin-kafka. We could also remove
'plugin' and make it incubator-metron-bro-kafka.
Also, I will submit a [MENTORS] discussion.
Jon
On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <ma...@apache.org> wrote:
> Browsing https://git.apache.org/ shows lots of examples. A few are quite
> prolific.
>
>
> On 4/5/17, 1:00 PM, "Nick Allen" <ni...@nickallen.org> wrote:
>
> Does anyone know any other Apache projects that are using multiple
> repos?
> I'd like to see what they've done just so we don't break convention.
>
>
>
>
>
>
> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > Yes, I will open an INFRA ticket. Just give me a little time to
> research
> > what we need.
> >
> > On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> >> Okay great, thanks. Would you mind throwing in an INFRA ticket for
> the
> >> new
> >> repo? I can take it all from there.
> >>
> >> Does anybody know if we have ASF resources to help answer the above
> legal
> >> question?
> >>
> >> Jon
> >>
> >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org>
> wrote:
> >>
> >> > (1) I am not sure if licensing is a problem here.
> >> >
> >> > (2) I am OK with whatever we need to get this effort done and
> under ASF.
> >> >
> >> >
> >> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
> >> wrote:
> >> >
> >> > > I'm working on this
> >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348>
> in
> >> > > preparation for the new repo and migration to a package. It
> looks
> >> like
> >> > in
> >> > > bro-plugins COPYING
> >> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
> >> > attributes
> >> > > to Nick, but in our version COPYING
> >> > > <https://github.com/apache/incubator-metron/tree/master/
> >> > > metron-sensors/bro-plugin-kafka/COPYING>
> >> > > points to the Apache License. Same with MAINTAINER (this
> >> > > <https://github.com/apache/incubator-metron/blob/master/
> >> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> >> > > vs this <
> https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
> >> INER
> >> > > >).
> >> > > I assume when we package this up and host it in Apache we need
> to
> >> give it
> >> > > the Apache license, and point to Metron for MAINTAINER. My
> questions
> >> > are:
> >> > >
> >> > > 1. Is there any legal/licensing concern here? I am taking
> changes
> >> from
> >> > > the bro-plugins version and pulling it into the Apache-hosted
> code.
> >> > IANAL
> >> > > 2. Nick - are you OK with these changes?
> >> > >
> >> > > Jon
> >> > >
> >> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> zeolla@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Can someone on the PMC submit a ticket to INFRA? It looks
> like
> >> > > > <https://www.apache.org/dev/infra-contact> committers aren't
> >> supposed
> >> > > to.
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> zeolla@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > I would be happy to try it again but I attempted to do that
> before
> >> with
> >> > > > bro packages and it failed to be able to handle it. I also
> tried
> >> using
> >> > > > branches of a repo with bro but that similarly failed (and
> was a
> >> pretty
> >> > > bad
> >> > > > idea to start with).
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org>
> wrote:
> >> > > >
> >> > > > We should be able to request just one alternate repo from
> INFRA, and
> >> > put
> >> > > a
> >> > > > top hierarchical level in it that doesn’t include a maven
> pom. As
> >> far
> >> > as
> >> > > > maven and clients are concerned, it
> >> > > >
> >> > > > just increases by 1 the path length to the root of the repo.
> >> > > >
> >> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com>
> wrote:
> >> > > >
> >> > > > Once we agree on a repo location to host this, I would be
> happy
> >> to
> >> > > put
> >> > > > together the package and update our environments to use
> bro-pkg
> >> to
> >> > > > install
> >> > > > the plugin. I have created METRON-813
> >> > > > <https://issues.apache.org/jira/browse/METRON-813> to
> track
> >> this
> >> > and
> >> > > > changed METRON-348 <
> >> > https://issues.apache.org/jira/browse/METRON-348
> >> > > >
> >> > > > to be
> >> > > > a sub-task.
> >> > > >
> >> > > > Otto - the bro packages model doesn't allow colocation
> with
> >> > anything
> >> > > > else.
> >> > > > That said, if we have two similar situations, and given
> the
> >> INFRA
> >> > > > example
> >> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
> >> linked to
> >> > > > before
> >> > > > was requesting 9 repos, perhaps we just request two repos.
> >> Would
> >> > > > someone
> >> > > > else mind putting that request in?
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> >> > > ottobackwards@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > Could we create a separate repo for more than on thing?
> like
> >> put …
> >> > > um
> >> > > > let’s say
> >> > > > a maven plugin and the bro plugin?
> >> > > >
> >> > > >
> >> > > >
> >> > > > On March 31, 2017 at 12:30:25, Nick Allen (
> nick@nickallen.org)
> >> > > wrote:
> >> > > >
> >> > > > I agree with everything that I've read.
> >> > > >
> >> > > > One of the guys from Bro had contacted me a while back,
> letting
> >> me
> >> > > know
> >> > > > that the packaging mechanism in Bro was ready for public
> >> > > consumption. I
> >> > > > just have not had cycles to do anything with it yet. They
> are
> >> not
> >> > > > wanting
> >> > > > to host any of the plugins.
> >> > > >
> >> > > > I thought the package mechanism requires that a package
> live
> >> within
> >> > > > its own
> >> > > > repo (which Casey confirmed). This put me in a bind on
> how to
> >> > tackle
> >> > > > this. I don't want to personally host the plugin in my own
> >> Github
> >> > > > repo. I
> >> > > > would prefer that we host it in a community repo; either
> Bro or
> >> > > Metron.
> >> > > > Since Bro is moving away from hosting their own plugins,
> that
> >> > leaves
> >> > > > Metron.
> >> > > >
> >> > > > It would be great if we could create a separate repo for
> the
> >> > plugin.
> >> > > > That
> >> > > > solves the challenge of using the packaging mechanism.
> >> > > >
> >> > > > We do need to reconcile what is in bro/bro-plugins and
> what is
> >> in
> >> > > > Metron.
> >> > > > There are some enhancements that I and others have made
> that
> >> never
> >> > > > made it
> >> > > > back into Metron. They never made it back, because the
> original
> >> > plan
> >> > > > was
> >> > > > just to switch to using the plugin from bro/bro-plugins
> before
> >> the
> >> > > > idea of
> >> > > > a packaging mechanism hit Bro. Reconciling should be
> fairly
> >> easy to
> >> > > > see by
> >> > > > just doing a diff.
> >> > > >
> >> > > > It would be great if others want to take on any of that
> work. I
> >> > would
> >> > > > be
> >> > > > glad to offer any support that you need. Thanks, Jon!
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> >> > zeolla@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Ok, great.
> >> > > > >
> >> > > > > I agree, I definitely want to hear from Nick on the
> topic. My
> >> > team
> >> > > is
> >> > > > > currently looking into enhancing the plugin as well to
> >> > potentially
> >> > > > allow
> >> > > > > sending to multiple clusters, investigating some issues
> we see
> >> > when
> >> > > > our
> >> > > > bro
> >> > > > > cluster is under load, turn it into a package, etc.
> >> > > > >
> >> > > > > The work you just did was on our to do list as well so
> I'm
> >> very
> >> > > > excited
> >> > > > to
> >> > > > > see it come through.
> >> > > > >
> >> > > > > Jon
> >> > > > >
> >> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> >> cestella@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > I *think* it's possible. People do ask for mirrors of
> >> directories
> >> > > > from
> >> > > > > time to time (see
> >> > https://issues.apache.org/jira/browse/INFRA-7060
> >> > > ).
> >> > > > If
> >> > > > > we
> >> > > > > think this is a good idea, we can pose it to INFRA as a
> >> request.
> >> > > I'd
> >> > > > love
> >> > > > > to see us be able to use the bro packaging
> infrastructure and
> >> get
> >> > > > more
> >> > > > > visibility for the plugin.
> >> > > > >
> >> > > > > I'd be particularly interested in Nick's opinion on
> this,
> >> though.
> >> > > > >
> >> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> >> > > zeolla@gmail.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > You can version packages -
> >> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> >> > > > > package.html#package-
> >> > > > > > versioning
> >> > > > > >
> >> > > > > > I agree that having a separate repo provided by Apache
> >> would be
> >> > > > optimal,
> >> > > > > I
> >> > > > > > just don't know the process for that or if it was even
> >> > reasonable
> >> > > > to
> >> > > > > > suggest.
> >> > > > > >
> >> > > > > > Jon
> >> > > > > >
> >> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> >> > cestella@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Looking at the bro packages, it appears that bro is
> >> expecting
> >> > > > things
> >> > > > to
> >> > > > > > be
> >> > > > > > > its own git repository. I wonder if we could either
> >> request
> >> > > INFRA
> >> > > > > > provide
> >> > > > > > > another repo for the bro-kafka plugin and integrate
> it
> >> into
> >> > > > metron as
> >> > > > a
> >> > > > > > git
> >> > > > > > > submodule *or* if we could request INFRA to create a
> >> github
> >> > > > mirror of
> >> > > > > the
> >> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not
> sure
> >> how
> >> > > > viable
> >> > > > > > either
> >> > > > > > > of those options are, frankly.
> >> > > > > > >
> >> > > > > > > One thing that I didn't see is how do you specify a
> >> > particular
> >> > > > release
> >> > > > > of
> >> > > > > > > the plugin that you want to install? For us, we'd
> want to
> >> > > > release the
> >> > > > > > > plugin along with the product. I didn't quite see
> how
> >> you'd
> >> > > push
> >> > > > > > releases
> >> > > > > > > for bro plugins.
> >> > > > > > >
> >> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> >> > > > cestella@gmail.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > So, I do agree with the concern. Is there a way
> to host
> >> the
> >> > > > package
> >> > > > > > > > within Metron? I definitely would like to see the
> >> > > > modifications at
> >> > > > > > > > https://github.com/bro/bro-plu
> >> gins/commit/b9f1f35415cb0db
> >> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into
> Metron and
> >> I'd
> >> > > > love for
> >> > > > > us
> >> > > > > > to
> >> > > > > > > > host the plugin.
> >> > > > > > > >
> >> > > > > > > > Thoughts?
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com
> <
> >> > > > zeolla@gmail.com>
> >> > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Today I was taking a look at METRON-812
> >> > > > > > > >> <
> https://issues.apache.org/jira/browse/METRON-812>,
> >> which
> >> > > > made me
> >> > > > > > > recall
> >> > > > > > > >> some conversations from a while back regarding
> where
> >> the
> >> > bro
> >> > > > kafka
> >> > > > > > > plugin
> >> > > > > > > >> should ultimately live, and how to update it.
> >> > > > > > > >>
> >> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
> >> > > > > jira/browse/METRON-348>
> >> > > > > > I
> >> > > > > > > >> brought up the fact that some important changes
> >> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> >> > > b9f1f35415cb0db06
> >> > > > > > > >> 5348da0a5043a8353b4a0a8>
> >> > > > > > > >> were made to the externally hosted version of
> the kafka
> >> > > > plugin,
> >> > > > and
> >> > > > > > were
> >> > > > > > > >> never introduced to Metron's hosted version
> (i.e. the
> >> one
> >> > we
> >> > > > use
> >> > > > > > > >> <https://github.com/apache/inc
> >> ubator-metron/blob/master/
> >> > > metr
> >> > > > > > > >>
> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> >> > > > > > > >> in vagrant when bro is installed). The
> conversation
> >> went
> >> > > down
> >> > > > the
> >> > > > > > route
> >> > > > > > > >> of
> >> > > > > > > >> discussing whether or not the bro kafka plugin
> code
> >> should
> >> > > > continue
> >> > > > > to
> >> > > > > > > >> live
> >> > > > > > > >> in Metron in the first place. Now, with
> METRON-812, I
> >> see
> >> > us
> >> > > > > further
> >> > > > > > > >> muddying the waters of where to go for the right
> >> plugin,
> >> > as
> >> > > > our
> >> > > > > > version
> >> > > > > > > is
> >> > > > > > > >> still missing the public changes but adds some
> very
> >> > > important
> >> > > > new
> >> > > > > > > >> functionality.
> >> > > > > > > >>
> >> > > > > > > >> I'd like to bring up the idea of using bro's
> packages
> >> > > > > > > >> <https://github.com/bro/packages> framework,
> released
> >> in
> >> > > > late 2016
> >> > > > > > > >> <
> >> > > >
> http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> >> > > > > > > >> (additional
> >> > > > > > > >> documentation here <
> >> > > > > > >
> http://bro-package-manager.readthedocs.io/en/stable/
> >> > > > > > > >> >),
> >> > > > > > > >> as a potential place for this to be
> hosted/referenced.
> >> > This
> >> > > > is a
> >> > > > > > simple
> >> > > > > > > >> and supported method (funded by Mozilla
> >> > > > > > > >> <https://blog.mozilla.org/blog
> >> /2015/12/10/mozilla-open-
> >> > > sourc
> >> > > > > > > >> e-support-first-awards-made/>)
> >> > > > > > > >> to install and uninstall bro scripts, plugins,
> etc.,
> >> and
> >> > it
> >> > > > also
> >> > > > > > allows
> >> > > > > > > us
> >> > > > > > > >> to continue to have enough control over updates
> to the
> >> > > plugin
> >> > > > so
> >> > > > > that
> >> > > > > > it
> >> > > > > > > >> will not slow down Metron development by having
> it as a
> >> > > > dependency
> >> > > > > > > >> (resolving both of Casey's concerns noted here
> >> > > > > > > >> <
> https://issues.apache.org/jira/browse/METRON-348?
> >> > > focusedCom
> >> > > > > > > >>
> mentId=15391865&page=com.atlassian.jira.plugin.system.
> >> > > > > > > >>
> issuetabpanels:comment-tabpanel#comment-15391865>,
> >> > > > > > > >> and I think this solution is supported by Nick's
> >> comments
> >> > > here
> >> > > > > > > >> <
> https://issues.apache.org/jira/browse/METRON-348?
> >> > > focusedCom
> >> > > > > > > >>
> mentId=15391872&page=com.atlassian.jira.plugin.system.
> >> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> >> > > > > > > >> as
> >> > > > > > > >> well).
> >> > > > > > > >>
> >> > > > > > > >> The only thing I'm not sure about is where to
> host the
> >> > > plugin
> >> > > > itself
> >> > > > > -
> >> > > > > > > my
> >> > > > > > > >> first thought would be Nick's github <
> >> > > > https://github.com/nickwallen
> >> > > > > >,
> >> > > > > > > as
> >> > > > > > > >> he
> >> > > > > > > >> really kicked off this effort, but maybe we can
> think
> >> of
> >> > > > something
> >> > > > > > > better.
> >> > > > > > > >>
> >> > > > > > > >> Is this approach of interest to anybody? It is
> >> extremely
> >> > > > simple to
> >> > > > > > put
> >> > > > > > > >> together - I was able to throw one together
> >> > > > > > > >> <https://github.com/bro/packages/blob/master/
> >> > > > > jonzeolla/bro-pkg.index>
> >> > > > > > > and
> >> > > > > > > >> get it working with a fresh bro 2.5 install when
> >> attending
> >> > > the
> >> > > > > brocon
> >> > > > > > > talk
> >> > > > > > > >> <
> >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> >> > > > > > > >> bro-packagemanager>
> >> > > > > > > >> (recording <https://www.youtube.com/watch
> >> ?v=9RFfPJeGkcE>,
> >> > > > slides
> >> > > > > > > >> <
> https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> >> > that
> >> > > > > introduced
> >> > > > > > > >> this
> >> > > > > > > >> to me in the first place.
> >> > > > > > > >>
> >> > > > > > > >> Jon
> >> > > > > > > >> --
> >> > > > > > > >>
> >> > > > > > > >> Jon
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > --
> >> > > > > >
> >> > > > > > Jon
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > > Jon
> >> > > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > --
> >> > >
> >> > > Jon
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
>
>
> --
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Matt Foley <ma...@apache.org>.
Browsing https://git.apache.org/ shows lots of examples. A few are quite prolific.
On 4/5/17, 1:00 PM, "Nick Allen" <ni...@nickallen.org> wrote:
Does anyone know any other Apache projects that are using multiple repos?
I'd like to see what they've done just so we don't break convention.
On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
> Yes, I will open an INFRA ticket. Just give me a little time to research
> what we need.
>
> On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Okay great, thanks. Would you mind throwing in an INFRA ticket for the
>> new
>> repo? I can take it all from there.
>>
>> Does anybody know if we have ASF resources to help answer the above legal
>> question?
>>
>> Jon
>>
>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
>>
>> > (1) I am not sure if licensing is a problem here.
>> >
>> > (2) I am OK with whatever we need to get this effort done and under ASF.
>> >
>> >
>> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>> >
>> > > I'm working on this
>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
>> > > preparation for the new repo and migration to a package. It looks
>> like
>> > in
>> > > bro-plugins COPYING
>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>> > attributes
>> > > to Nick, but in our version COPYING
>> > > <https://github.com/apache/incubator-metron/tree/master/
>> > > metron-sensors/bro-plugin-kafka/COPYING>
>> > > points to the Apache License. Same with MAINTAINER (this
>> > > <https://github.com/apache/incubator-metron/blob/master/
>> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>> INER
>> > > >).
>> > > I assume when we package this up and host it in Apache we need to
>> give it
>> > > the Apache license, and point to Metron for MAINTAINER. My questions
>> > are:
>> > >
>> > > 1. Is there any legal/licensing concern here? I am taking changes
>> from
>> > > the bro-plugins version and pulling it into the Apache-hosted code.
>> > IANAL
>> > > 2. Nick - are you OK with these changes?
>> > >
>> > > Jon
>> > >
>> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
>> > wrote:
>> > >
>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like
>> > > > <https://www.apache.org/dev/infra-contact> committers aren't
>> supposed
>> > > to.
>> > > >
>> > > > Jon
>> > > >
>> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com>
>> > > wrote:
>> > > >
>> > > > I would be happy to try it again but I attempted to do that before
>> with
>> > > > bro packages and it failed to be able to handle it. I also tried
>> using
>> > > > branches of a repo with bro but that similarly failed (and was a
>> pretty
>> > > bad
>> > > > idea to start with).
>> > > >
>> > > > Jon
>> > > >
>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
>> > > >
>> > > > We should be able to request just one alternate repo from INFRA, and
>> > put
>> > > a
>> > > > top hierarchical level in it that doesn’t include a maven pom. As
>> far
>> > as
>> > > > maven and clients are concerned, it
>> > > >
>> > > > just increases by 1 the path length to the root of the repo.
>> > > >
>> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>> > > >
>> > > > Once we agree on a repo location to host this, I would be happy
>> to
>> > > put
>> > > > together the package and update our environments to use bro-pkg
>> to
>> > > > install
>> > > > the plugin. I have created METRON-813
>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track
>> this
>> > and
>> > > > changed METRON-348 <
>> > https://issues.apache.org/jira/browse/METRON-348
>> > > >
>> > > > to be
>> > > > a sub-task.
>> > > >
>> > > > Otto - the bro packages model doesn't allow colocation with
>> > anything
>> > > > else.
>> > > > That said, if we have two similar situations, and given the
>> INFRA
>> > > > example
>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
>> linked to
>> > > > before
>> > > > was requesting 9 repos, perhaps we just request two repos.
>> Would
>> > > > someone
>> > > > else mind putting that request in?
>> > > >
>> > > > Jon
>> > > >
>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>> > > ottobackwards@gmail.com>
>> > > > wrote:
>> > > >
>> > > > Could we create a separate repo for more than on thing? like
>> put …
>> > > um
>> > > > let’s say
>> > > > a maven plugin and the bro plugin?
>> > > >
>> > > >
>> > > >
>> > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
>> > > wrote:
>> > > >
>> > > > I agree with everything that I've read.
>> > > >
>> > > > One of the guys from Bro had contacted me a while back, letting
>> me
>> > > know
>> > > > that the packaging mechanism in Bro was ready for public
>> > > consumption. I
>> > > > just have not had cycles to do anything with it yet. They are
>> not
>> > > > wanting
>> > > > to host any of the plugins.
>> > > >
>> > > > I thought the package mechanism requires that a package live
>> within
>> > > > its own
>> > > > repo (which Casey confirmed). This put me in a bind on how to
>> > tackle
>> > > > this. I don't want to personally host the plugin in my own
>> Github
>> > > > repo. I
>> > > > would prefer that we host it in a community repo; either Bro or
>> > > Metron.
>> > > > Since Bro is moving away from hosting their own plugins, that
>> > leaves
>> > > > Metron.
>> > > >
>> > > > It would be great if we could create a separate repo for the
>> > plugin.
>> > > > That
>> > > > solves the challenge of using the packaging mechanism.
>> > > >
>> > > > We do need to reconcile what is in bro/bro-plugins and what is
>> in
>> > > > Metron.
>> > > > There are some enhancements that I and others have made that
>> never
>> > > > made it
>> > > > back into Metron. They never made it back, because the original
>> > plan
>> > > > was
>> > > > just to switch to using the plugin from bro/bro-plugins before
>> the
>> > > > idea of
>> > > > a packaging mechanism hit Bro. Reconciling should be fairly
>> easy to
>> > > > see by
>> > > > just doing a diff.
>> > > >
>> > > > It would be great if others want to take on any of that work. I
>> > would
>> > > > be
>> > > > glad to offer any support that you need. Thanks, Jon!
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>> > zeolla@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Ok, great.
>> > > > >
>> > > > > I agree, I definitely want to hear from Nick on the topic. My
>> > team
>> > > is
>> > > > > currently looking into enhancing the plugin as well to
>> > potentially
>> > > > allow
>> > > > > sending to multiple clusters, investigating some issues we see
>> > when
>> > > > our
>> > > > bro
>> > > > > cluster is under load, turn it into a package, etc.
>> > > > >
>> > > > > The work you just did was on our to do list as well so I'm
>> very
>> > > > excited
>> > > > to
>> > > > > see it come through.
>> > > > >
>> > > > > Jon
>> > > > >
>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>> cestella@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > I *think* it's possible. People do ask for mirrors of
>> directories
>> > > > from
>> > > > > time to time (see
>> > https://issues.apache.org/jira/browse/INFRA-7060
>> > > ).
>> > > > If
>> > > > > we
>> > > > > think this is a good idea, we can pose it to INFRA as a
>> request.
>> > > I'd
>> > > > love
>> > > > > to see us be able to use the bro packaging infrastructure and
>> get
>> > > > more
>> > > > > visibility for the plugin.
>> > > > >
>> > > > > I'd be particularly interested in Nick's opinion on this,
>> though.
>> > > > >
>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
>> > > zeolla@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > You can version packages -
>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>> > > > > package.html#package-
>> > > > > > versioning
>> > > > > >
>> > > > > > I agree that having a separate repo provided by Apache
>> would be
>> > > > optimal,
>> > > > > I
>> > > > > > just don't know the process for that or if it was even
>> > reasonable
>> > > > to
>> > > > > > suggest.
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
>> > cestella@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > Looking at the bro packages, it appears that bro is
>> expecting
>> > > > things
>> > > > to
>> > > > > > be
>> > > > > > > its own git repository. I wonder if we could either
>> request
>> > > INFRA
>> > > > > > provide
>> > > > > > > another repo for the bro-kafka plugin and integrate it
>> into
>> > > > metron as
>> > > > a
>> > > > > > git
>> > > > > > > submodule *or* if we could request INFRA to create a
>> github
>> > > > mirror of
>> > > > > the
>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
>> how
>> > > > viable
>> > > > > > either
>> > > > > > > of those options are, frankly.
>> > > > > > >
>> > > > > > > One thing that I didn't see is how do you specify a
>> > particular
>> > > > release
>> > > > > of
>> > > > > > > the plugin that you want to install? For us, we'd want to
>> > > > release the
>> > > > > > > plugin along with the product. I didn't quite see how
>> you'd
>> > > push
>> > > > > > releases
>> > > > > > > for bro plugins.
>> > > > > > >
>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
>> > > > cestella@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > So, I do agree with the concern. Is there a way to host
>> the
>> > > > package
>> > > > > > > > within Metron? I definitely would like to see the
>> > > > modifications at
>> > > > > > > > https://github.com/bro/bro-plu
>> gins/commit/b9f1f35415cb0db
>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
>> I'd
>> > > > love for
>> > > > > us
>> > > > > > to
>> > > > > > > > host the plugin.
>> > > > > > > >
>> > > > > > > > Thoughts?
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
>> > > > zeolla@gmail.com>
>> > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Today I was taking a look at METRON-812
>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
>> which
>> > > > made me
>> > > > > > > recall
>> > > > > > > >> some conversations from a while back regarding where
>> the
>> > bro
>> > > > kafka
>> > > > > > > plugin
>> > > > > > > >> should ultimately live, and how to update it.
>> > > > > > > >>
>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
>> > > > > jira/browse/METRON-348>
>> > > > > > I
>> > > > > > > >> brought up the fact that some important changes
>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
>> > > b9f1f35415cb0db06
>> > > > > > > >> 5348da0a5043a8353b4a0a8>
>> > > > > > > >> were made to the externally hosted version of the kafka
>> > > > plugin,
>> > > > and
>> > > > > > were
>> > > > > > > >> never introduced to Metron's hosted version (i.e. the
>> one
>> > we
>> > > > use
>> > > > > > > >> <https://github.com/apache/inc
>> ubator-metron/blob/master/
>> > > metr
>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>> > > > > > > >> in vagrant when bro is installed). The conversation
>> went
>> > > down
>> > > > the
>> > > > > > route
>> > > > > > > >> of
>> > > > > > > >> discussing whether or not the bro kafka plugin code
>> should
>> > > > continue
>> > > > > to
>> > > > > > > >> live
>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I
>> see
>> > us
>> > > > > further
>> > > > > > > >> muddying the waters of where to go for the right
>> plugin,
>> > as
>> > > > our
>> > > > > > version
>> > > > > > > is
>> > > > > > > >> still missing the public changes but adds some very
>> > > important
>> > > > new
>> > > > > > > >> functionality.
>> > > > > > > >>
>> > > > > > > >> I'd like to bring up the idea of using bro's packages
>> > > > > > > >> <https://github.com/bro/packages> framework, released
>> in
>> > > > late 2016
>> > > > > > > >> <
>> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>> > > > > > > >> (additional
>> > > > > > > >> documentation here <
>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>> > > > > > > >> >),
>> > > > > > > >> as a potential place for this to be hosted/referenced.
>> > This
>> > > > is a
>> > > > > > simple
>> > > > > > > >> and supported method (funded by Mozilla
>> > > > > > > >> <https://blog.mozilla.org/blog
>> /2015/12/10/mozilla-open-
>> > > sourc
>> > > > > > > >> e-support-first-awards-made/>)
>> > > > > > > >> to install and uninstall bro scripts, plugins, etc.,
>> and
>> > it
>> > > > also
>> > > > > > allows
>> > > > > > > us
>> > > > > > > >> to continue to have enough control over updates to the
>> > > plugin
>> > > > so
>> > > > > that
>> > > > > > it
>> > > > > > > >> will not slow down Metron development by having it as a
>> > > > dependency
>> > > > > > > >> (resolving both of Casey's concerns noted here
>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>> > > focusedCom
>> > > > > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
>> > > > > > > >> and I think this solution is supported by Nick's
>> comments
>> > > here
>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>> > > focusedCom
>> > > > > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>> > > > > > > >> as
>> > > > > > > >> well).
>> > > > > > > >>
>> > > > > > > >> The only thing I'm not sure about is where to host the
>> > > plugin
>> > > > itself
>> > > > > -
>> > > > > > > my
>> > > > > > > >> first thought would be Nick's github <
>> > > > https://github.com/nickwallen
>> > > > > >,
>> > > > > > > as
>> > > > > > > >> he
>> > > > > > > >> really kicked off this effort, but maybe we can think
>> of
>> > > > something
>> > > > > > > better.
>> > > > > > > >>
>> > > > > > > >> Is this approach of interest to anybody? It is
>> extremely
>> > > > simple to
>> > > > > > put
>> > > > > > > >> together - I was able to throw one together
>> > > > > > > >> <https://github.com/bro/packages/blob/master/
>> > > > > jonzeolla/bro-pkg.index>
>> > > > > > > and
>> > > > > > > >> get it working with a fresh bro 2.5 install when
>> attending
>> > > the
>> > > > > brocon
>> > > > > > > talk
>> > > > > > > >> <
>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>> > > > > > > >> bro-packagemanager>
>> > > > > > > >> (recording <https://www.youtube.com/watch
>> ?v=9RFfPJeGkcE>,
>> > > > slides
>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>> > that
>> > > > > introduced
>> > > > > > > >> this
>> > > > > > > >> to me in the first place.
>> > > > > > > >>
>> > > > > > > >> Jon
>> > > > > > > >> --
>> > > > > > > >>
>> > > > > > > >> Jon
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > --
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Jon
>> > > > >
>> > > >
>> > > > --
>> > > >
>> > > > Jon
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Jon
>> > > >
>> > > > --
>> > > >
>> > > > Jon
>> > > >
>> > > --
>> > >
>> > > Jon
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] The bro kafka plugin
Posted by Nick Allen <ni...@nickallen.org>.
Does anyone know any other Apache projects that are using multiple repos?
I'd like to see what they've done just so we don't break convention.
On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <ni...@nickallen.org> wrote:
> Yes, I will open an INFRA ticket. Just give me a little time to research
> what we need.
>
> On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Okay great, thanks. Would you mind throwing in an INFRA ticket for the
>> new
>> repo? I can take it all from there.
>>
>> Does anybody know if we have ASF resources to help answer the above legal
>> question?
>>
>> Jon
>>
>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
>>
>> > (1) I am not sure if licensing is a problem here.
>> >
>> > (2) I am OK with whatever we need to get this effort done and under ASF.
>> >
>> >
>> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>> >
>> > > I'm working on this
>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
>> > > preparation for the new repo and migration to a package. It looks
>> like
>> > in
>> > > bro-plugins COPYING
>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>> > attributes
>> > > to Nick, but in our version COPYING
>> > > <https://github.com/apache/incubator-metron/tree/master/
>> > > metron-sensors/bro-plugin-kafka/COPYING>
>> > > points to the Apache License. Same with MAINTAINER (this
>> > > <https://github.com/apache/incubator-metron/blob/master/
>> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>> INER
>> > > >).
>> > > I assume when we package this up and host it in Apache we need to
>> give it
>> > > the Apache license, and point to Metron for MAINTAINER. My questions
>> > are:
>> > >
>> > > 1. Is there any legal/licensing concern here? I am taking changes
>> from
>> > > the bro-plugins version and pulling it into the Apache-hosted code.
>> > IANAL
>> > > 2. Nick - are you OK with these changes?
>> > >
>> > > Jon
>> > >
>> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
>> > wrote:
>> > >
>> > > > Can someone on the PMC submit a ticket to INFRA? It looks like
>> > > > <https://www.apache.org/dev/infra-contact> committers aren't
>> supposed
>> > > to.
>> > > >
>> > > > Jon
>> > > >
>> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com>
>> > > wrote:
>> > > >
>> > > > I would be happy to try it again but I attempted to do that before
>> with
>> > > > bro packages and it failed to be able to handle it. I also tried
>> using
>> > > > branches of a repo with bro but that similarly failed (and was a
>> pretty
>> > > bad
>> > > > idea to start with).
>> > > >
>> > > > Jon
>> > > >
>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
>> > > >
>> > > > We should be able to request just one alternate repo from INFRA, and
>> > put
>> > > a
>> > > > top hierarchical level in it that doesn’t include a maven pom. As
>> far
>> > as
>> > > > maven and clients are concerned, it
>> > > >
>> > > > just increases by 1 the path length to the root of the repo.
>> > > >
>> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>> > > >
>> > > > Once we agree on a repo location to host this, I would be happy
>> to
>> > > put
>> > > > together the package and update our environments to use bro-pkg
>> to
>> > > > install
>> > > > the plugin. I have created METRON-813
>> > > > <https://issues.apache.org/jira/browse/METRON-813> to track
>> this
>> > and
>> > > > changed METRON-348 <
>> > https://issues.apache.org/jira/browse/METRON-348
>> > > >
>> > > > to be
>> > > > a sub-task.
>> > > >
>> > > > Otto - the bro packages model doesn't allow colocation with
>> > anything
>> > > > else.
>> > > > That said, if we have two similar situations, and given the
>> INFRA
>> > > > example
>> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey
>> linked to
>> > > > before
>> > > > was requesting 9 repos, perhaps we just request two repos.
>> Would
>> > > > someone
>> > > > else mind putting that request in?
>> > > >
>> > > > Jon
>> > > >
>> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>> > > ottobackwards@gmail.com>
>> > > > wrote:
>> > > >
>> > > > Could we create a separate repo for more than on thing? like
>> put …
>> > > um
>> > > > let’s say
>> > > > a maven plugin and the bro plugin?
>> > > >
>> > > >
>> > > >
>> > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
>> > > wrote:
>> > > >
>> > > > I agree with everything that I've read.
>> > > >
>> > > > One of the guys from Bro had contacted me a while back, letting
>> me
>> > > know
>> > > > that the packaging mechanism in Bro was ready for public
>> > > consumption. I
>> > > > just have not had cycles to do anything with it yet. They are
>> not
>> > > > wanting
>> > > > to host any of the plugins.
>> > > >
>> > > > I thought the package mechanism requires that a package live
>> within
>> > > > its own
>> > > > repo (which Casey confirmed). This put me in a bind on how to
>> > tackle
>> > > > this. I don't want to personally host the plugin in my own
>> Github
>> > > > repo. I
>> > > > would prefer that we host it in a community repo; either Bro or
>> > > Metron.
>> > > > Since Bro is moving away from hosting their own plugins, that
>> > leaves
>> > > > Metron.
>> > > >
>> > > > It would be great if we could create a separate repo for the
>> > plugin.
>> > > > That
>> > > > solves the challenge of using the packaging mechanism.
>> > > >
>> > > > We do need to reconcile what is in bro/bro-plugins and what is
>> in
>> > > > Metron.
>> > > > There are some enhancements that I and others have made that
>> never
>> > > > made it
>> > > > back into Metron. They never made it back, because the original
>> > plan
>> > > > was
>> > > > just to switch to using the plugin from bro/bro-plugins before
>> the
>> > > > idea of
>> > > > a packaging mechanism hit Bro. Reconciling should be fairly
>> easy to
>> > > > see by
>> > > > just doing a diff.
>> > > >
>> > > > It would be great if others want to take on any of that work. I
>> > would
>> > > > be
>> > > > glad to offer any support that you need. Thanks, Jon!
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>> > zeolla@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Ok, great.
>> > > > >
>> > > > > I agree, I definitely want to hear from Nick on the topic. My
>> > team
>> > > is
>> > > > > currently looking into enhancing the plugin as well to
>> > potentially
>> > > > allow
>> > > > > sending to multiple clusters, investigating some issues we see
>> > when
>> > > > our
>> > > > bro
>> > > > > cluster is under load, turn it into a package, etc.
>> > > > >
>> > > > > The work you just did was on our to do list as well so I'm
>> very
>> > > > excited
>> > > > to
>> > > > > see it come through.
>> > > > >
>> > > > > Jon
>> > > > >
>> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>> cestella@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > I *think* it's possible. People do ask for mirrors of
>> directories
>> > > > from
>> > > > > time to time (see
>> > https://issues.apache.org/jira/browse/INFRA-7060
>> > > ).
>> > > > If
>> > > > > we
>> > > > > think this is a good idea, we can pose it to INFRA as a
>> request.
>> > > I'd
>> > > > love
>> > > > > to see us be able to use the bro packaging infrastructure and
>> get
>> > > > more
>> > > > > visibility for the plugin.
>> > > > >
>> > > > > I'd be particularly interested in Nick's opinion on this,
>> though.
>> > > > >
>> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
>> > > zeolla@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > You can version packages -
>> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>> > > > > package.html#package-
>> > > > > > versioning
>> > > > > >
>> > > > > > I agree that having a separate repo provided by Apache
>> would be
>> > > > optimal,
>> > > > > I
>> > > > > > just don't know the process for that or if it was even
>> > reasonable
>> > > > to
>> > > > > > suggest.
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
>> > cestella@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > Looking at the bro packages, it appears that bro is
>> expecting
>> > > > things
>> > > > to
>> > > > > > be
>> > > > > > > its own git repository. I wonder if we could either
>> request
>> > > INFRA
>> > > > > > provide
>> > > > > > > another repo for the bro-kafka plugin and integrate it
>> into
>> > > > metron as
>> > > > a
>> > > > > > git
>> > > > > > > submodule *or* if we could request INFRA to create a
>> github
>> > > > mirror of
>> > > > > the
>> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
>> how
>> > > > viable
>> > > > > > either
>> > > > > > > of those options are, frankly.
>> > > > > > >
>> > > > > > > One thing that I didn't see is how do you specify a
>> > particular
>> > > > release
>> > > > > of
>> > > > > > > the plugin that you want to install? For us, we'd want to
>> > > > release the
>> > > > > > > plugin along with the product. I didn't quite see how
>> you'd
>> > > push
>> > > > > > releases
>> > > > > > > for bro plugins.
>> > > > > > >
>> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
>> > > > cestella@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > So, I do agree with the concern. Is there a way to host
>> the
>> > > > package
>> > > > > > > > within Metron? I definitely would like to see the
>> > > > modifications at
>> > > > > > > > https://github.com/bro/bro-plu
>> gins/commit/b9f1f35415cb0db
>> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
>> I'd
>> > > > love for
>> > > > > us
>> > > > > > to
>> > > > > > > > host the plugin.
>> > > > > > > >
>> > > > > > > > Thoughts?
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
>> > > > zeolla@gmail.com>
>> > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Today I was taking a look at METRON-812
>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
>> which
>> > > > made me
>> > > > > > > recall
>> > > > > > > >> some conversations from a while back regarding where
>> the
>> > bro
>> > > > kafka
>> > > > > > > plugin
>> > > > > > > >> should ultimately live, and how to update it.
>> > > > > > > >>
>> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
>> > > > > jira/browse/METRON-348>
>> > > > > > I
>> > > > > > > >> brought up the fact that some important changes
>> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
>> > > b9f1f35415cb0db06
>> > > > > > > >> 5348da0a5043a8353b4a0a8>
>> > > > > > > >> were made to the externally hosted version of the kafka
>> > > > plugin,
>> > > > and
>> > > > > > were
>> > > > > > > >> never introduced to Metron's hosted version (i.e. the
>> one
>> > we
>> > > > use
>> > > > > > > >> <https://github.com/apache/inc
>> ubator-metron/blob/master/
>> > > metr
>> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>> > > > > > > >> in vagrant when bro is installed). The conversation
>> went
>> > > down
>> > > > the
>> > > > > > route
>> > > > > > > >> of
>> > > > > > > >> discussing whether or not the bro kafka plugin code
>> should
>> > > > continue
>> > > > > to
>> > > > > > > >> live
>> > > > > > > >> in Metron in the first place. Now, with METRON-812, I
>> see
>> > us
>> > > > > further
>> > > > > > > >> muddying the waters of where to go for the right
>> plugin,
>> > as
>> > > > our
>> > > > > > version
>> > > > > > > is
>> > > > > > > >> still missing the public changes but adds some very
>> > > important
>> > > > new
>> > > > > > > >> functionality.
>> > > > > > > >>
>> > > > > > > >> I'd like to bring up the idea of using bro's packages
>> > > > > > > >> <https://github.com/bro/packages> framework, released
>> in
>> > > > late 2016
>> > > > > > > >> <
>> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>> > > > > > > >> (additional
>> > > > > > > >> documentation here <
>> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>> > > > > > > >> >),
>> > > > > > > >> as a potential place for this to be hosted/referenced.
>> > This
>> > > > is a
>> > > > > > simple
>> > > > > > > >> and supported method (funded by Mozilla
>> > > > > > > >> <https://blog.mozilla.org/blog
>> /2015/12/10/mozilla-open-
>> > > sourc
>> > > > > > > >> e-support-first-awards-made/>)
>> > > > > > > >> to install and uninstall bro scripts, plugins, etc.,
>> and
>> > it
>> > > > also
>> > > > > > allows
>> > > > > > > us
>> > > > > > > >> to continue to have enough control over updates to the
>> > > plugin
>> > > > so
>> > > > > that
>> > > > > > it
>> > > > > > > >> will not slow down Metron development by having it as a
>> > > > dependency
>> > > > > > > >> (resolving both of Casey's concerns noted here
>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>> > > focusedCom
>> > > > > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
>> > > > > > > >> and I think this solution is supported by Nick's
>> comments
>> > > here
>> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>> > > focusedCom
>> > > > > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
>> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>> > > > > > > >> as
>> > > > > > > >> well).
>> > > > > > > >>
>> > > > > > > >> The only thing I'm not sure about is where to host the
>> > > plugin
>> > > > itself
>> > > > > -
>> > > > > > > my
>> > > > > > > >> first thought would be Nick's github <
>> > > > https://github.com/nickwallen
>> > > > > >,
>> > > > > > > as
>> > > > > > > >> he
>> > > > > > > >> really kicked off this effort, but maybe we can think
>> of
>> > > > something
>> > > > > > > better.
>> > > > > > > >>
>> > > > > > > >> Is this approach of interest to anybody? It is
>> extremely
>> > > > simple to
>> > > > > > put
>> > > > > > > >> together - I was able to throw one together
>> > > > > > > >> <https://github.com/bro/packages/blob/master/
>> > > > > jonzeolla/bro-pkg.index>
>> > > > > > > and
>> > > > > > > >> get it working with a fresh bro 2.5 install when
>> attending
>> > > the
>> > > > > brocon
>> > > > > > > talk
>> > > > > > > >> <
>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>> > > > > > > >> bro-packagemanager>
>> > > > > > > >> (recording <https://www.youtube.com/watch
>> ?v=9RFfPJeGkcE>,
>> > > > slides
>> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>> > that
>> > > > > introduced
>> > > > > > > >> this
>> > > > > > > >> to me in the first place.
>> > > > > > > >>
>> > > > > > > >> Jon
>> > > > > > > >> --
>> > > > > > > >>
>> > > > > > > >> Jon
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > --
>> > > > > >
>> > > > > > Jon
>> > > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Jon
>> > > > >
>> > > >
>> > > > --
>> > > >
>> > > > Jon
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Jon
>> > > >
>> > > > --
>> > > >
>> > > > Jon
>> > > >
>> > > --
>> > >
>> > > Jon
>> > >
>> >
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] The bro kafka plugin
Posted by Nick Allen <ni...@nickallen.org>.
Yes, I will open an INFRA ticket. Just give me a little time to research
what we need.
On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Okay great, thanks. Would you mind throwing in an INFRA ticket for the new
> repo? I can take it all from there.
>
> Does anybody know if we have ASF resources to help answer the above legal
> question?
>
> Jon
>
> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
>
> > (1) I am not sure if licensing is a problem here.
> >
> > (2) I am OK with whatever we need to get this effort done and under ASF.
> >
> >
> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> > > I'm working on this
> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
> > > preparation for the new repo and migration to a package. It looks like
> > in
> > > bro-plugins COPYING
> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
> > attributes
> > > to Nick, but in our version COPYING
> > > <https://github.com/apache/incubator-metron/tree/master/
> > > metron-sensors/bro-plugin-kafka/COPYING>
> > > points to the Apache License. Same with MAINTAINER (this
> > > <https://github.com/apache/incubator-metron/blob/master/
> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/
> MAINTAINER
> > > >).
> > > I assume when we package this up and host it in Apache we need to give
> it
> > > the Apache license, and point to Metron for MAINTAINER. My questions
> > are:
> > >
> > > 1. Is there any legal/licensing concern here? I am taking changes
> from
> > > the bro-plugins version and pulling it into the Apache-hosted code.
> > IANAL
> > > 2. Nick - are you OK with these changes?
> > >
> > > Jon
> > >
> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> > >
> > > > Can someone on the PMC submit a ticket to INFRA? It looks like
> > > > <https://www.apache.org/dev/infra-contact> committers aren't
> supposed
> > > to.
> > > >
> > > > Jon
> > > >
> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > > >
> > > > I would be happy to try it again but I attempted to do that before
> with
> > > > bro packages and it failed to be able to handle it. I also tried
> using
> > > > branches of a repo with bro but that similarly failed (and was a
> pretty
> > > bad
> > > > idea to start with).
> > > >
> > > > Jon
> > > >
> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
> > > >
> > > > We should be able to request just one alternate repo from INFRA, and
> > put
> > > a
> > > > top hierarchical level in it that doesn’t include a maven pom. As
> far
> > as
> > > > maven and clients are concerned, it
> > > >
> > > > just increases by 1 the path length to the root of the repo.
> > > >
> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> > > >
> > > > Once we agree on a repo location to host this, I would be happy
> to
> > > put
> > > > together the package and update our environments to use bro-pkg
> to
> > > > install
> > > > the plugin. I have created METRON-813
> > > > <https://issues.apache.org/jira/browse/METRON-813> to track this
> > and
> > > > changed METRON-348 <
> > https://issues.apache.org/jira/browse/METRON-348
> > > >
> > > > to be
> > > > a sub-task.
> > > >
> > > > Otto - the bro packages model doesn't allow colocation with
> > anything
> > > > else.
> > > > That said, if we have two similar situations, and given the INFRA
> > > > example
> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked
> to
> > > > before
> > > > was requesting 9 repos, perhaps we just request two repos. Would
> > > > someone
> > > > else mind putting that request in?
> > > >
> > > > Jon
> > > >
> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > > wrote:
> > > >
> > > > Could we create a separate repo for more than on thing? like
> put …
> > > um
> > > > let’s say
> > > > a maven plugin and the bro plugin?
> > > >
> > > >
> > > >
> > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
> > > wrote:
> > > >
> > > > I agree with everything that I've read.
> > > >
> > > > One of the guys from Bro had contacted me a while back, letting
> me
> > > know
> > > > that the packaging mechanism in Bro was ready for public
> > > consumption. I
> > > > just have not had cycles to do anything with it yet. They are not
> > > > wanting
> > > > to host any of the plugins.
> > > >
> > > > I thought the package mechanism requires that a package live
> within
> > > > its own
> > > > repo (which Casey confirmed). This put me in a bind on how to
> > tackle
> > > > this. I don't want to personally host the plugin in my own Github
> > > > repo. I
> > > > would prefer that we host it in a community repo; either Bro or
> > > Metron.
> > > > Since Bro is moving away from hosting their own plugins, that
> > leaves
> > > > Metron.
> > > >
> > > > It would be great if we could create a separate repo for the
> > plugin.
> > > > That
> > > > solves the challenge of using the packaging mechanism.
> > > >
> > > > We do need to reconcile what is in bro/bro-plugins and what is in
> > > > Metron.
> > > > There are some enhancements that I and others have made that
> never
> > > > made it
> > > > back into Metron. They never made it back, because the original
> > plan
> > > > was
> > > > just to switch to using the plugin from bro/bro-plugins before
> the
> > > > idea of
> > > > a packaging mechanism hit Bro. Reconciling should be fairly easy
> to
> > > > see by
> > > > just doing a diff.
> > > >
> > > > It would be great if others want to take on any of that work. I
> > would
> > > > be
> > > > glad to offer any support that you need. Thanks, Jon!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> > zeolla@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Ok, great.
> > > > >
> > > > > I agree, I definitely want to hear from Nick on the topic. My
> > team
> > > is
> > > > > currently looking into enhancing the plugin as well to
> > potentially
> > > > allow
> > > > > sending to multiple clusters, investigating some issues we see
> > when
> > > > our
> > > > bro
> > > > > cluster is under load, turn it into a package, etc.
> > > > >
> > > > > The work you just did was on our to do list as well so I'm very
> > > > excited
> > > > to
> > > > > see it come through.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> cestella@gmail.com>
> > > > wrote:
> > > > >
> > > > > I *think* it's possible. People do ask for mirrors of
> directories
> > > > from
> > > > > time to time (see
> > https://issues.apache.org/jira/browse/INFRA-7060
> > > ).
> > > > If
> > > > > we
> > > > > think this is a good idea, we can pose it to INFRA as a
> request.
> > > I'd
> > > > love
> > > > > to see us be able to use the bro packaging infrastructure and
> get
> > > > more
> > > > > visibility for the plugin.
> > > > >
> > > > > I'd be particularly interested in Nick's opinion on this,
> though.
> > > > >
> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > You can version packages -
> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > > package.html#package-
> > > > > > versioning
> > > > > >
> > > > > > I agree that having a separate repo provided by Apache would
> be
> > > > optimal,
> > > > > I
> > > > > > just don't know the process for that or if it was even
> > reasonable
> > > > to
> > > > > > suggest.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> > cestella@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Looking at the bro packages, it appears that bro is
> expecting
> > > > things
> > > > to
> > > > > > be
> > > > > > > its own git repository. I wonder if we could either request
> > > INFRA
> > > > > > provide
> > > > > > > another repo for the bro-kafka plugin and integrate it into
> > > > metron as
> > > > a
> > > > > > git
> > > > > > > submodule *or* if we could request INFRA to create a github
> > > > mirror of
> > > > > the
> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
> how
> > > > viable
> > > > > > either
> > > > > > > of those options are, frankly.
> > > > > > >
> > > > > > > One thing that I didn't see is how do you specify a
> > particular
> > > > release
> > > > > of
> > > > > > > the plugin that you want to install? For us, we'd want to
> > > > release the
> > > > > > > plugin along with the product. I didn't quite see how you'd
> > > push
> > > > > > releases
> > > > > > > for bro plugins.
> > > > > > >
> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> > > > cestella@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > So, I do agree with the concern. Is there a way to host
> the
> > > > package
> > > > > > > > within Metron? I definitely would like to see the
> > > > modifications at
> > > > > > > > https://github.com/bro/bro-
> plugins/commit/b9f1f35415cb0db
> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
> I'd
> > > > love for
> > > > > us
> > > > > > to
> > > > > > > > host the plugin.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> > > > zeolla@gmail.com>
> > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Today I was taking a look at METRON-812
> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
> which
> > > > made me
> > > > > > > recall
> > > > > > > >> some conversations from a while back regarding where the
> > bro
> > > > kafka
> > > > > > > plugin
> > > > > > > >> should ultimately live, and how to update it.
> > > > > > > >>
> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
> > > > > jira/browse/METRON-348>
> > > > > > I
> > > > > > > >> brought up the fact that some important changes
> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> > > b9f1f35415cb0db06
> > > > > > > >> 5348da0a5043a8353b4a0a8>
> > > > > > > >> were made to the externally hosted version of the kafka
> > > > plugin,
> > > > and
> > > > > > were
> > > > > > > >> never introduced to Metron's hosted version (i.e. the
> one
> > we
> > > > use
> > > > > > > >> <https://github.com/apache/
> incubator-metron/blob/master/
> > > metr
> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > > > > >> in vagrant when bro is installed). The conversation went
> > > down
> > > > the
> > > > > > route
> > > > > > > >> of
> > > > > > > >> discussing whether or not the bro kafka plugin code
> should
> > > > continue
> > > > > to
> > > > > > > >> live
> > > > > > > >> in Metron in the first place. Now, with METRON-812, I
> see
> > us
> > > > > further
> > > > > > > >> muddying the waters of where to go for the right plugin,
> > as
> > > > our
> > > > > > version
> > > > > > > is
> > > > > > > >> still missing the public changes but adds some very
> > > important
> > > > new
> > > > > > > >> functionality.
> > > > > > > >>
> > > > > > > >> I'd like to bring up the idea of using bro's packages
> > > > > > > >> <https://github.com/bro/packages> framework, released
> in
> > > > late 2016
> > > > > > > >> <
> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > > > > >> (additional
> > > > > > > >> documentation here <
> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > > > > >> >),
> > > > > > > >> as a potential place for this to be hosted/referenced.
> > This
> > > > is a
> > > > > > simple
> > > > > > > >> and supported method (funded by Mozilla
> > > > > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-
> > > sourc
> > > > > > > >> e-support-first-awards-made/>)
> > > > > > > >> to install and uninstall bro scripts, plugins, etc., and
> > it
> > > > also
> > > > > > allows
> > > > > > > us
> > > > > > > >> to continue to have enough control over updates to the
> > > plugin
> > > > so
> > > > > that
> > > > > > it
> > > > > > > >> will not slow down Metron development by having it as a
> > > > dependency
> > > > > > > >> (resolving both of Casey's concerns noted here
> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> > > focusedCom
> > > > > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > > > > >> and I think this solution is supported by Nick's
> comments
> > > here
> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> > > focusedCom
> > > > > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > > > > >> as
> > > > > > > >> well).
> > > > > > > >>
> > > > > > > >> The only thing I'm not sure about is where to host the
> > > plugin
> > > > itself
> > > > > -
> > > > > > > my
> > > > > > > >> first thought would be Nick's github <
> > > > https://github.com/nickwallen
> > > > > >,
> > > > > > > as
> > > > > > > >> he
> > > > > > > >> really kicked off this effort, but maybe we can think of
> > > > something
> > > > > > > better.
> > > > > > > >>
> > > > > > > >> Is this approach of interest to anybody? It is extremely
> > > > simple to
> > > > > > put
> > > > > > > >> together - I was able to throw one together
> > > > > > > >> <https://github.com/bro/packages/blob/master/
> > > > > jonzeolla/bro-pkg.index>
> > > > > > > and
> > > > > > > >> get it working with a fresh bro 2.5 install when
> attending
> > > the
> > > > > brocon
> > > > > > > talk
> > > > > > > >> <
> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > > > > >> bro-packagemanager>
> > > > > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE
> >,
> > > > slides
> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> > that
> > > > > introduced
> > > > > > > >> this
> > > > > > > >> to me in the first place.
> > > > > > > >>
> > > > > > > >> Jon
> > > > > > > >> --
> > > > > > > >>
> > > > > > > >> Jon
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > >
> > > > > > Jon
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jon
> > > > >
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by Casey Stella <ce...@gmail.com>.
IANAL, but to my knowledge, the bro-kafka plugin hosted inside of the bro
project and the bro-kafka plugin hosted inside of Metron are two separate
plugins. Indeed, these two plugins have diverged in capabilities. Nick
contributed them both to two separate OSS projects, which is totally fine.
If you want to submit a PR to correct the multithreading bug that exists in
the Metron bro-kafka plugin, then I think that's perfectly fine. I
wouldn't think of it as making them equal, but rather correcting a bug in
the Metron bro-kafka plugin.
It might be good to pose the question on dev with the subject prefaced with
[MENTORS]. They will likely have different opinions.
One bit of detail not mentioned yet is that bro-kafka plugin inside of bro
is 3-clause BSD licensed. The question in my mind is if we submit this
correction whether it falls under bundling (which would necessitate a line
in our LICENSE file), is a derivative work (which is kosher because it's
BSD licensed) or whether it's just the contribution of a piece of
functionality that another piece of software has.
Casey
On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Okay great, thanks. Would you mind throwing in an INFRA ticket for the new
> repo? I can take it all from there.
>
> Does anybody know if we have ASF resources to help answer the above legal
> question?
>
> Jon
>
> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
>
> > (1) I am not sure if licensing is a problem here.
> >
> > (2) I am OK with whatever we need to get this effort done and under ASF.
> >
> >
> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> > > I'm working on this
> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
> > > preparation for the new repo and migration to a package. It looks like
> > in
> > > bro-plugins COPYING
> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
> > attributes
> > > to Nick, but in our version COPYING
> > > <https://github.com/apache/incubator-metron/tree/master/
> > > metron-sensors/bro-plugin-kafka/COPYING>
> > > points to the Apache License. Same with MAINTAINER (this
> > > <https://github.com/apache/incubator-metron/blob/master/
> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/
> MAINTAINER
> > > >).
> > > I assume when we package this up and host it in Apache we need to give
> it
> > > the Apache license, and point to Metron for MAINTAINER. My questions
> > are:
> > >
> > > 1. Is there any legal/licensing concern here? I am taking changes
> from
> > > the bro-plugins version and pulling it into the Apache-hosted code.
> > IANAL
> > > 2. Nick - are you OK with these changes?
> > >
> > > Jon
> > >
> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> > >
> > > > Can someone on the PMC submit a ticket to INFRA? It looks like
> > > > <https://www.apache.org/dev/infra-contact> committers aren't
> supposed
> > > to.
> > > >
> > > > Jon
> > > >
> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > > >
> > > > I would be happy to try it again but I attempted to do that before
> with
> > > > bro packages and it failed to be able to handle it. I also tried
> using
> > > > branches of a repo with bro but that similarly failed (and was a
> pretty
> > > bad
> > > > idea to start with).
> > > >
> > > > Jon
> > > >
> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
> > > >
> > > > We should be able to request just one alternate repo from INFRA, and
> > put
> > > a
> > > > top hierarchical level in it that doesn’t include a maven pom. As
> far
> > as
> > > > maven and clients are concerned, it
> > > >
> > > > just increases by 1 the path length to the root of the repo.
> > > >
> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> > > >
> > > > Once we agree on a repo location to host this, I would be happy
> to
> > > put
> > > > together the package and update our environments to use bro-pkg
> to
> > > > install
> > > > the plugin. I have created METRON-813
> > > > <https://issues.apache.org/jira/browse/METRON-813> to track this
> > and
> > > > changed METRON-348 <
> > https://issues.apache.org/jira/browse/METRON-348
> > > >
> > > > to be
> > > > a sub-task.
> > > >
> > > > Otto - the bro packages model doesn't allow colocation with
> > anything
> > > > else.
> > > > That said, if we have two similar situations, and given the INFRA
> > > > example
> > > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked
> to
> > > > before
> > > > was requesting 9 repos, perhaps we just request two repos. Would
> > > > someone
> > > > else mind putting that request in?
> > > >
> > > > Jon
> > > >
> > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> > > ottobackwards@gmail.com>
> > > > wrote:
> > > >
> > > > Could we create a separate repo for more than on thing? like
> put …
> > > um
> > > > let’s say
> > > > a maven plugin and the bro plugin?
> > > >
> > > >
> > > >
> > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
> > > wrote:
> > > >
> > > > I agree with everything that I've read.
> > > >
> > > > One of the guys from Bro had contacted me a while back, letting
> me
> > > know
> > > > that the packaging mechanism in Bro was ready for public
> > > consumption. I
> > > > just have not had cycles to do anything with it yet. They are not
> > > > wanting
> > > > to host any of the plugins.
> > > >
> > > > I thought the package mechanism requires that a package live
> within
> > > > its own
> > > > repo (which Casey confirmed). This put me in a bind on how to
> > tackle
> > > > this. I don't want to personally host the plugin in my own Github
> > > > repo. I
> > > > would prefer that we host it in a community repo; either Bro or
> > > Metron.
> > > > Since Bro is moving away from hosting their own plugins, that
> > leaves
> > > > Metron.
> > > >
> > > > It would be great if we could create a separate repo for the
> > plugin.
> > > > That
> > > > solves the challenge of using the packaging mechanism.
> > > >
> > > > We do need to reconcile what is in bro/bro-plugins and what is in
> > > > Metron.
> > > > There are some enhancements that I and others have made that
> never
> > > > made it
> > > > back into Metron. They never made it back, because the original
> > plan
> > > > was
> > > > just to switch to using the plugin from bro/bro-plugins before
> the
> > > > idea of
> > > > a packaging mechanism hit Bro. Reconciling should be fairly easy
> to
> > > > see by
> > > > just doing a diff.
> > > >
> > > > It would be great if others want to take on any of that work. I
> > would
> > > > be
> > > > glad to offer any support that you need. Thanks, Jon!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> > zeolla@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Ok, great.
> > > > >
> > > > > I agree, I definitely want to hear from Nick on the topic. My
> > team
> > > is
> > > > > currently looking into enhancing the plugin as well to
> > potentially
> > > > allow
> > > > > sending to multiple clusters, investigating some issues we see
> > when
> > > > our
> > > > bro
> > > > > cluster is under load, turn it into a package, etc.
> > > > >
> > > > > The work you just did was on our to do list as well so I'm very
> > > > excited
> > > > to
> > > > > see it come through.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
> cestella@gmail.com>
> > > > wrote:
> > > > >
> > > > > I *think* it's possible. People do ask for mirrors of
> directories
> > > > from
> > > > > time to time (see
> > https://issues.apache.org/jira/browse/INFRA-7060
> > > ).
> > > > If
> > > > > we
> > > > > think this is a good idea, we can pose it to INFRA as a
> request.
> > > I'd
> > > > love
> > > > > to see us be able to use the bro packaging infrastructure and
> get
> > > > more
> > > > > visibility for the plugin.
> > > > >
> > > > > I'd be particularly interested in Nick's opinion on this,
> though.
> > > > >
> > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > You can version packages -
> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > > package.html#package-
> > > > > > versioning
> > > > > >
> > > > > > I agree that having a separate repo provided by Apache would
> be
> > > > optimal,
> > > > > I
> > > > > > just don't know the process for that or if it was even
> > reasonable
> > > > to
> > > > > > suggest.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> > cestella@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Looking at the bro packages, it appears that bro is
> expecting
> > > > things
> > > > to
> > > > > > be
> > > > > > > its own git repository. I wonder if we could either request
> > > INFRA
> > > > > > provide
> > > > > > > another repo for the bro-kafka plugin and integrate it into
> > > > metron as
> > > > a
> > > > > > git
> > > > > > > submodule *or* if we could request INFRA to create a github
> > > > mirror of
> > > > > the
> > > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure
> how
> > > > viable
> > > > > > either
> > > > > > > of those options are, frankly.
> > > > > > >
> > > > > > > One thing that I didn't see is how do you specify a
> > particular
> > > > release
> > > > > of
> > > > > > > the plugin that you want to install? For us, we'd want to
> > > > release the
> > > > > > > plugin along with the product. I didn't quite see how you'd
> > > push
> > > > > > releases
> > > > > > > for bro plugins.
> > > > > > >
> > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> > > > cestella@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > So, I do agree with the concern. Is there a way to host
> the
> > > > package
> > > > > > > > within Metron? I definitely would like to see the
> > > > modifications at
> > > > > > > > https://github.com/bro/bro-
> plugins/commit/b9f1f35415cb0db
> > > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and
> I'd
> > > > love for
> > > > > us
> > > > > > to
> > > > > > > > host the plugin.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> > > > zeolla@gmail.com>
> > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Today I was taking a look at METRON-812
> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
> which
> > > > made me
> > > > > > > recall
> > > > > > > >> some conversations from a while back regarding where the
> > bro
> > > > kafka
> > > > > > > plugin
> > > > > > > >> should ultimately live, and how to update it.
> > > > > > > >>
> > > > > > > >> Back in METRON-348 <https://issues.apache.org/
> > > > > jira/browse/METRON-348>
> > > > > > I
> > > > > > > >> brought up the fact that some important changes
> > > > > > > >> <https://github.com/bro/bro-plugins/commit/
> > > b9f1f35415cb0db06
> > > > > > > >> 5348da0a5043a8353b4a0a8>
> > > > > > > >> were made to the externally hosted version of the kafka
> > > > plugin,
> > > > and
> > > > > > were
> > > > > > > >> never introduced to Metron's hosted version (i.e. the
> one
> > we
> > > > use
> > > > > > > >> <https://github.com/apache/
> incubator-metron/blob/master/
> > > metr
> > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > > > > >> in vagrant when bro is installed). The conversation went
> > > down
> > > > the
> > > > > > route
> > > > > > > >> of
> > > > > > > >> discussing whether or not the bro kafka plugin code
> should
> > > > continue
> > > > > to
> > > > > > > >> live
> > > > > > > >> in Metron in the first place. Now, with METRON-812, I
> see
> > us
> > > > > further
> > > > > > > >> muddying the waters of where to go for the right plugin,
> > as
> > > > our
> > > > > > version
> > > > > > > is
> > > > > > > >> still missing the public changes but adds some very
> > > important
> > > > new
> > > > > > > >> functionality.
> > > > > > > >>
> > > > > > > >> I'd like to bring up the idea of using bro's packages
> > > > > > > >> <https://github.com/bro/packages> framework, released
> in
> > > > late 2016
> > > > > > > >> <
> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > > > > >> (additional
> > > > > > > >> documentation here <
> > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > > > > >> >),
> > > > > > > >> as a potential place for this to be hosted/referenced.
> > This
> > > > is a
> > > > > > simple
> > > > > > > >> and supported method (funded by Mozilla
> > > > > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-
> > > sourc
> > > > > > > >> e-support-first-awards-made/>)
> > > > > > > >> to install and uninstall bro scripts, plugins, etc., and
> > it
> > > > also
> > > > > > allows
> > > > > > > us
> > > > > > > >> to continue to have enough control over updates to the
> > > plugin
> > > > so
> > > > > that
> > > > > > it
> > > > > > > >> will not slow down Metron development by having it as a
> > > > dependency
> > > > > > > >> (resolving both of Casey's concerns noted here
> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> > > focusedCom
> > > > > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > > > > >> and I think this solution is supported by Nick's
> comments
> > > here
> > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> > > focusedCom
> > > > > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > > > > >> as
> > > > > > > >> well).
> > > > > > > >>
> > > > > > > >> The only thing I'm not sure about is where to host the
> > > plugin
> > > > itself
> > > > > -
> > > > > > > my
> > > > > > > >> first thought would be Nick's github <
> > > > https://github.com/nickwallen
> > > > > >,
> > > > > > > as
> > > > > > > >> he
> > > > > > > >> really kicked off this effort, but maybe we can think of
> > > > something
> > > > > > > better.
> > > > > > > >>
> > > > > > > >> Is this approach of interest to anybody? It is extremely
> > > > simple to
> > > > > > put
> > > > > > > >> together - I was able to throw one together
> > > > > > > >> <https://github.com/bro/packages/blob/master/
> > > > > jonzeolla/bro-pkg.index>
> > > > > > > and
> > > > > > > >> get it working with a fresh bro 2.5 install when
> attending
> > > the
> > > > > brocon
> > > > > > > talk
> > > > > > > >> <
> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > > > > >> bro-packagemanager>
> > > > > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE
> >,
> > > > slides
> > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> > that
> > > > > introduced
> > > > > > > >> this
> > > > > > > >> to me in the first place.
> > > > > > > >>
> > > > > > > >> Jon
> > > > > > > >> --
> > > > > > > >>
> > > > > > > >> Jon
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > >
> > > > > > Jon
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jon
> > > > >
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Okay great, thanks. Would you mind throwing in an INFRA ticket for the new
repo? I can take it all from there.
Does anybody know if we have ASF resources to help answer the above legal
question?
Jon
On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <ni...@nickallen.org> wrote:
> (1) I am not sure if licensing is a problem here.
>
> (2) I am OK with whatever we need to get this effort done and under ASF.
>
>
> On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > I'm working on this
> > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
> > preparation for the new repo and migration to a package. It looks like
> in
> > bro-plugins COPYING
> > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
> attributes
> > to Nick, but in our version COPYING
> > <https://github.com/apache/incubator-metron/tree/master/
> > metron-sensors/bro-plugin-kafka/COPYING>
> > points to the Apache License. Same with MAINTAINER (this
> > <https://github.com/apache/incubator-metron/blob/master/
> > metron-sensors/bro-plugin-kafka/MAINTAINER>
> > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTAINER
> > >).
> > I assume when we package this up and host it in Apache we need to give it
> > the Apache license, and point to Metron for MAINTAINER. My questions
> are:
> >
> > 1. Is there any legal/licensing concern here? I am taking changes from
> > the bro-plugins version and pulling it into the Apache-hosted code.
> IANAL
> > 2. Nick - are you OK with these changes?
> >
> > Jon
> >
> > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> > > Can someone on the PMC submit a ticket to INFRA? It looks like
> > > <https://www.apache.org/dev/infra-contact> committers aren't supposed
> > to.
> > >
> > > Jon
> > >
> > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> > >
> > > I would be happy to try it again but I attempted to do that before with
> > > bro packages and it failed to be able to handle it. I also tried using
> > > branches of a repo with bro but that similarly failed (and was a pretty
> > bad
> > > idea to start with).
> > >
> > > Jon
> > >
> > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
> > >
> > > We should be able to request just one alternate repo from INFRA, and
> put
> > a
> > > top hierarchical level in it that doesn’t include a maven pom. As far
> as
> > > maven and clients are concerned, it
> > >
> > > just increases by 1 the path length to the root of the repo.
> > >
> > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> > >
> > > Once we agree on a repo location to host this, I would be happy to
> > put
> > > together the package and update our environments to use bro-pkg to
> > > install
> > > the plugin. I have created METRON-813
> > > <https://issues.apache.org/jira/browse/METRON-813> to track this
> and
> > > changed METRON-348 <
> https://issues.apache.org/jira/browse/METRON-348
> > >
> > > to be
> > > a sub-task.
> > >
> > > Otto - the bro packages model doesn't allow colocation with
> anything
> > > else.
> > > That said, if we have two similar situations, and given the INFRA
> > > example
> > > <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to
> > > before
> > > was requesting 9 repos, perhaps we just request two repos. Would
> > > someone
> > > else mind putting that request in?
> > >
> > > Jon
> > >
> > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > > wrote:
> > >
> > > Could we create a separate repo for more than on thing? like put …
> > um
> > > let’s say
> > > a maven plugin and the bro plugin?
> > >
> > >
> > >
> > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
> > wrote:
> > >
> > > I agree with everything that I've read.
> > >
> > > One of the guys from Bro had contacted me a while back, letting me
> > know
> > > that the packaging mechanism in Bro was ready for public
> > consumption. I
> > > just have not had cycles to do anything with it yet. They are not
> > > wanting
> > > to host any of the plugins.
> > >
> > > I thought the package mechanism requires that a package live within
> > > its own
> > > repo (which Casey confirmed). This put me in a bind on how to
> tackle
> > > this. I don't want to personally host the plugin in my own Github
> > > repo. I
> > > would prefer that we host it in a community repo; either Bro or
> > Metron.
> > > Since Bro is moving away from hosting their own plugins, that
> leaves
> > > Metron.
> > >
> > > It would be great if we could create a separate repo for the
> plugin.
> > > That
> > > solves the challenge of using the packaging mechanism.
> > >
> > > We do need to reconcile what is in bro/bro-plugins and what is in
> > > Metron.
> > > There are some enhancements that I and others have made that never
> > > made it
> > > back into Metron. They never made it back, because the original
> plan
> > > was
> > > just to switch to using the plugin from bro/bro-plugins before the
> > > idea of
> > > a packaging mechanism hit Bro. Reconciling should be fairly easy to
> > > see by
> > > just doing a diff.
> > >
> > > It would be great if others want to take on any of that work. I
> would
> > > be
> > > glad to offer any support that you need. Thanks, Jon!
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
> zeolla@gmail.com
> > >
> > > wrote:
> > >
> > > > Ok, great.
> > > >
> > > > I agree, I definitely want to hear from Nick on the topic. My
> team
> > is
> > > > currently looking into enhancing the plugin as well to
> potentially
> > > allow
> > > > sending to multiple clusters, investigating some issues we see
> when
> > > our
> > > bro
> > > > cluster is under load, turn it into a package, etc.
> > > >
> > > > The work you just did was on our to do list as well so I'm very
> > > excited
> > > to
> > > > see it come through.
> > > >
> > > > Jon
> > > >
> > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com>
> > > wrote:
> > > >
> > > > I *think* it's possible. People do ask for mirrors of directories
> > > from
> > > > time to time (see
> https://issues.apache.org/jira/browse/INFRA-7060
> > ).
> > > If
> > > > we
> > > > think this is a good idea, we can pose it to INFRA as a request.
> > I'd
> > > love
> > > > to see us be able to use the bro packaging infrastructure and get
> > > more
> > > > visibility for the plugin.
> > > >
> > > > I'd be particularly interested in Nick's opinion on this, though.
> > > >
> > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> > zeolla@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > You can version packages -
> > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > package.html#package-
> > > > > versioning
> > > > >
> > > > > I agree that having a separate repo provided by Apache would be
> > > optimal,
> > > > I
> > > > > just don't know the process for that or if it was even
> reasonable
> > > to
> > > > > suggest.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
> cestella@gmail.com>
> > > wrote:
> > > > >
> > > > > > Looking at the bro packages, it appears that bro is expecting
> > > things
> > > to
> > > > > be
> > > > > > its own git repository. I wonder if we could either request
> > INFRA
> > > > > provide
> > > > > > another repo for the bro-kafka plugin and integrate it into
> > > metron as
> > > a
> > > > > git
> > > > > > submodule *or* if we could request INFRA to create a github
> > > mirror of
> > > > the
> > > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure how
> > > viable
> > > > > either
> > > > > > of those options are, frankly.
> > > > > >
> > > > > > One thing that I didn't see is how do you specify a
> particular
> > > release
> > > > of
> > > > > > the plugin that you want to install? For us, we'd want to
> > > release the
> > > > > > plugin along with the product. I didn't quite see how you'd
> > push
> > > > > releases
> > > > > > for bro plugins.
> > > > > >
> > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> > > cestella@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > So, I do agree with the concern. Is there a way to host the
> > > package
> > > > > > > within Metron? I definitely would like to see the
> > > modifications at
> > > > > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd
> > > love for
> > > > us
> > > > > to
> > > > > > > host the plugin.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com>
> > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Today I was taking a look at METRON-812
> > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>, which
> > > made me
> > > > > > recall
> > > > > > >> some conversations from a while back regarding where the
> bro
> > > kafka
> > > > > > plugin
> > > > > > >> should ultimately live, and how to update it.
> > > > > > >>
> > > > > > >> Back in METRON-348 <https://issues.apache.org/
> > > > jira/browse/METRON-348>
> > > > > I
> > > > > > >> brought up the fact that some important changes
> > > > > > >> <https://github.com/bro/bro-plugins/commit/
> > b9f1f35415cb0db06
> > > > > > >> 5348da0a5043a8353b4a0a8>
> > > > > > >> were made to the externally hosted version of the kafka
> > > plugin,
> > > and
> > > > > were
> > > > > > >> never introduced to Metron's hosted version (i.e. the one
> we
> > > use
> > > > > > >> <https://github.com/apache/incubator-metron/blob/master/
> > metr
> > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > > > >> in vagrant when bro is installed). The conversation went
> > down
> > > the
> > > > > route
> > > > > > >> of
> > > > > > >> discussing whether or not the bro kafka plugin code should
> > > continue
> > > > to
> > > > > > >> live
> > > > > > >> in Metron in the first place. Now, with METRON-812, I see
> us
> > > > further
> > > > > > >> muddying the waters of where to go for the right plugin,
> as
> > > our
> > > > > version
> > > > > > is
> > > > > > >> still missing the public changes but adds some very
> > important
> > > new
> > > > > > >> functionality.
> > > > > > >>
> > > > > > >> I'd like to bring up the idea of using bro's packages
> > > > > > >> <https://github.com/bro/packages> framework, released in
> > > late 2016
> > > > > > >> <
> > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > > > >> (additional
> > > > > > >> documentation here <
> > > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > > > >> >),
> > > > > > >> as a potential place for this to be hosted/referenced.
> This
> > > is a
> > > > > simple
> > > > > > >> and supported method (funded by Mozilla
> > > > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-
> > sourc
> > > > > > >> e-support-first-awards-made/>)
> > > > > > >> to install and uninstall bro scripts, plugins, etc., and
> it
> > > also
> > > > > allows
> > > > > > us
> > > > > > >> to continue to have enough control over updates to the
> > plugin
> > > so
> > > > that
> > > > > it
> > > > > > >> will not slow down Metron development by having it as a
> > > dependency
> > > > > > >> (resolving both of Casey's concerns noted here
> > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> > focusedCom
> > > > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > > > >> and I think this solution is supported by Nick's comments
> > here
> > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> > focusedCom
> > > > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > > > >> as
> > > > > > >> well).
> > > > > > >>
> > > > > > >> The only thing I'm not sure about is where to host the
> > plugin
> > > itself
> > > > -
> > > > > > my
> > > > > > >> first thought would be Nick's github <
> > > https://github.com/nickwallen
> > > > >,
> > > > > > as
> > > > > > >> he
> > > > > > >> really kicked off this effort, but maybe we can think of
> > > something
> > > > > > better.
> > > > > > >>
> > > > > > >> Is this approach of interest to anybody? It is extremely
> > > simple to
> > > > > put
> > > > > > >> together - I was able to throw one together
> > > > > > >> <https://github.com/bro/packages/blob/master/
> > > > jonzeolla/bro-pkg.index>
> > > > > > and
> > > > > > >> get it working with a fresh bro 2.5 install when attending
> > the
> > > > brocon
> > > > > > talk
> > > > > > >> <
> https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > > > >> bro-packagemanager>
> > > > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>,
> > > slides
> > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> that
> > > > introduced
> > > > > > >> this
> > > > > > >> to me in the first place.
> > > > > > >>
> > > > > > >> Jon
> > > > > > >> --
> > > > > > >>
> > > > > > >> Jon
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > >
> > > > > Jon
> > > > >
> > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> > > --
> > >
> > > Jon
> > >
> > >
> > >
> > > --
> > >
> > > Jon
> > >
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Nick Allen <ni...@nickallen.org>.
(1) I am not sure if licensing is a problem here.
(2) I am OK with whatever we need to get this effort done and under ASF.
On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> I'm working on this
> <https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
> preparation for the new repo and migration to a package. It looks like in
> bro-plugins COPYING
> <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING> attributes
> to Nick, but in our version COPYING
> <https://github.com/apache/incubator-metron/tree/master/
> metron-sensors/bro-plugin-kafka/COPYING>
> points to the Apache License. Same with MAINTAINER (this
> <https://github.com/apache/incubator-metron/blob/master/
> metron-sensors/bro-plugin-kafka/MAINTAINER>
> vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTAINER
> >).
> I assume when we package this up and host it in Apache we need to give it
> the Apache license, and point to Metron for MAINTAINER. My questions are:
>
> 1. Is there any legal/licensing concern here? I am taking changes from
> the bro-plugins version and pulling it into the Apache-hosted code. IANAL
> 2. Nick - are you OK with these changes?
>
> Jon
>
> On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > Can someone on the PMC submit a ticket to INFRA? It looks like
> > <https://www.apache.org/dev/infra-contact> committers aren't supposed
> to.
> >
> > Jon
> >
> > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >
> > I would be happy to try it again but I attempted to do that before with
> > bro packages and it failed to be able to handle it. I also tried using
> > branches of a repo with bro but that similarly failed (and was a pretty
> bad
> > idea to start with).
> >
> > Jon
> >
> > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
> >
> > We should be able to request just one alternate repo from INFRA, and put
> a
> > top hierarchical level in it that doesn’t include a maven pom. As far as
> > maven and clients are concerned, it
> >
> > just increases by 1 the path length to the root of the repo.
> >
> > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
> >
> > Once we agree on a repo location to host this, I would be happy to
> put
> > together the package and update our environments to use bro-pkg to
> > install
> > the plugin. I have created METRON-813
> > <https://issues.apache.org/jira/browse/METRON-813> to track this and
> > changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348
> >
> > to be
> > a sub-task.
> >
> > Otto - the bro packages model doesn't allow colocation with anything
> > else.
> > That said, if we have two similar situations, and given the INFRA
> > example
> > <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to
> > before
> > was requesting 9 repos, perhaps we just request two repos. Would
> > someone
> > else mind putting that request in?
> >
> > Jon
> >
> > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> ottobackwards@gmail.com>
> > wrote:
> >
> > Could we create a separate repo for more than on thing? like put …
> um
> > let’s say
> > a maven plugin and the bro plugin?
> >
> >
> >
> > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
> wrote:
> >
> > I agree with everything that I've read.
> >
> > One of the guys from Bro had contacted me a while back, letting me
> know
> > that the packaging mechanism in Bro was ready for public
> consumption. I
> > just have not had cycles to do anything with it yet. They are not
> > wanting
> > to host any of the plugins.
> >
> > I thought the package mechanism requires that a package live within
> > its own
> > repo (which Casey confirmed). This put me in a bind on how to tackle
> > this. I don't want to personally host the plugin in my own Github
> > repo. I
> > would prefer that we host it in a community repo; either Bro or
> Metron.
> > Since Bro is moving away from hosting their own plugins, that leaves
> > Metron.
> >
> > It would be great if we could create a separate repo for the plugin.
> > That
> > solves the challenge of using the packaging mechanism.
> >
> > We do need to reconcile what is in bro/bro-plugins and what is in
> > Metron.
> > There are some enhancements that I and others have made that never
> > made it
> > back into Metron. They never made it back, because the original plan
> > was
> > just to switch to using the plugin from bro/bro-plugins before the
> > idea of
> > a packaging mechanism hit Bro. Reconciling should be fairly easy to
> > see by
> > just doing a diff.
> >
> > It would be great if others want to take on any of that work. I would
> > be
> > glad to offer any support that you need. Thanks, Jon!
> >
> >
> >
> >
> >
> > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> >
> > > Ok, great.
> > >
> > > I agree, I definitely want to hear from Nick on the topic. My team
> is
> > > currently looking into enhancing the plugin as well to potentially
> > allow
> > > sending to multiple clusters, investigating some issues we see when
> > our
> > bro
> > > cluster is under load, turn it into a package, etc.
> > >
> > > The work you just did was on our to do list as well so I'm very
> > excited
> > to
> > > see it come through.
> > >
> > > Jon
> > >
> > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com>
> > wrote:
> > >
> > > I *think* it's possible. People do ask for mirrors of directories
> > from
> > > time to time (see https://issues.apache.org/jira/browse/INFRA-7060
> ).
> > If
> > > we
> > > think this is a good idea, we can pose it to INFRA as a request.
> I'd
> > love
> > > to see us be able to use the bro packaging infrastructure and get
> > more
> > > visibility for the plugin.
> > >
> > > I'd be particularly interested in Nick's opinion on this, though.
> > >
> > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
> zeolla@gmail.com
> > >
> > > wrote:
> > >
> > > > You can version packages -
> > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > package.html#package-
> > > > versioning
> > > >
> > > > I agree that having a separate repo provided by Apache would be
> > optimal,
> > > I
> > > > just don't know the process for that or if it was even reasonable
> > to
> > > > suggest.
> > > >
> > > > Jon
> > > >
> > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com>
> > wrote:
> > > >
> > > > > Looking at the bro packages, it appears that bro is expecting
> > things
> > to
> > > > be
> > > > > its own git repository. I wonder if we could either request
> INFRA
> > > > provide
> > > > > another repo for the bro-kafka plugin and integrate it into
> > metron as
> > a
> > > > git
> > > > > submodule *or* if we could request INFRA to create a github
> > mirror of
> > > the
> > > > > metron-sensors/bro-kafka-plugin directory. I'm not sure how
> > viable
> > > > either
> > > > > of those options are, frankly.
> > > > >
> > > > > One thing that I didn't see is how do you specify a particular
> > release
> > > of
> > > > > the plugin that you want to install? For us, we'd want to
> > release the
> > > > > plugin along with the product. I didn't quite see how you'd
> push
> > > > releases
> > > > > for bro plugins.
> > > > >
> > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> > cestella@gmail.com>
> > > > wrote:
> > > > >
> > > > > > So, I do agree with the concern. Is there a way to host the
> > package
> > > > > > within Metron? I definitely would like to see the
> > modifications at
> > > > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd
> > love for
> > > us
> > > > to
> > > > > > host the plugin.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> > zeolla@gmail.com>
> >
> > > > > > wrote:
> > > > > >
> > > > > >> Today I was taking a look at METRON-812
> > > > > >> <https://issues.apache.org/jira/browse/METRON-812>, which
> > made me
> > > > > recall
> > > > > >> some conversations from a while back regarding where the bro
> > kafka
> > > > > plugin
> > > > > >> should ultimately live, and how to update it.
> > > > > >>
> > > > > >> Back in METRON-348 <https://issues.apache.org/
> > > jira/browse/METRON-348>
> > > > I
> > > > > >> brought up the fact that some important changes
> > > > > >> <https://github.com/bro/bro-plugins/commit/
> b9f1f35415cb0db06
> > > > > >> 5348da0a5043a8353b4a0a8>
> > > > > >> were made to the externally hosted version of the kafka
> > plugin,
> > and
> > > > were
> > > > > >> never introduced to Metron's hosted version (i.e. the one we
> > use
> > > > > >> <https://github.com/apache/incubator-metron/blob/master/
> metr
> > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > > >> in vagrant when bro is installed). The conversation went
> down
> > the
> > > > route
> > > > > >> of
> > > > > >> discussing whether or not the bro kafka plugin code should
> > continue
> > > to
> > > > > >> live
> > > > > >> in Metron in the first place. Now, with METRON-812, I see us
> > > further
> > > > > >> muddying the waters of where to go for the right plugin, as
> > our
> > > > version
> > > > > is
> > > > > >> still missing the public changes but adds some very
> important
> > new
> > > > > >> functionality.
> > > > > >>
> > > > > >> I'd like to bring up the idea of using bro's packages
> > > > > >> <https://github.com/bro/packages> framework, released in
> > late 2016
> > > > > >> <
> > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > > >> (additional
> > > > > >> documentation here <
> > > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > > >> >),
> > > > > >> as a potential place for this to be hosted/referenced. This
> > is a
> > > > simple
> > > > > >> and supported method (funded by Mozilla
> > > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-
> sourc
> > > > > >> e-support-first-awards-made/>)
> > > > > >> to install and uninstall bro scripts, plugins, etc., and it
> > also
> > > > allows
> > > > > us
> > > > > >> to continue to have enough control over updates to the
> plugin
> > so
> > > that
> > > > it
> > > > > >> will not slow down Metron development by having it as a
> > dependency
> > > > > >> (resolving both of Casey's concerns noted here
> > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> focusedCom
> > > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > > >> and I think this solution is supported by Nick's comments
> here
> > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
> focusedCom
> > > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > > >> as
> > > > > >> well).
> > > > > >>
> > > > > >> The only thing I'm not sure about is where to host the
> plugin
> > itself
> > > -
> > > > > my
> > > > > >> first thought would be Nick's github <
> > https://github.com/nickwallen
> > > >,
> > > > > as
> > > > > >> he
> > > > > >> really kicked off this effort, but maybe we can think of
> > something
> > > > > better.
> > > > > >>
> > > > > >> Is this approach of interest to anybody? It is extremely
> > simple to
> > > > put
> > > > > >> together - I was able to throw one together
> > > > > >> <https://github.com/bro/packages/blob/master/
> > > jonzeolla/bro-pkg.index>
> > > > > and
> > > > > >> get it working with a fresh bro 2.5 install when attending
> the
> > > brocon
> > > > > talk
> > > > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > > >> bro-packagemanager>
> > > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>,
> > slides
> > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> > > introduced
> > > > > >> this
> > > > > >> to me in the first place.
> > > > > >>
> > > > > >> Jon
> > > > > >> --
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> > > --
> > >
> > > Jon
> > >
> >
> > --
> >
> > Jon
> >
> >
> >
> > --
> >
> > Jon
> >
> > --
> >
> > Jon
> >
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I'm working on this
<https://github.com/JonZeolla/incubator-metron/tree/METRON-348> in
preparation for the new repo and migration to a package. It looks like in
bro-plugins COPYING
<https://github.com/bro/bro-plugins/blob/master/kafka/COPYING> attributes
to Nick, but in our version COPYING
<https://github.com/apache/incubator-metron/tree/master/metron-sensors/bro-plugin-kafka/COPYING>
points to the Apache License. Same with MAINTAINER (this
<https://github.com/apache/incubator-metron/blob/master/metron-sensors/bro-plugin-kafka/MAINTAINER>
vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTAINER>).
I assume when we package this up and host it in Apache we need to give it
the Apache license, and point to Metron for MAINTAINER. My questions are:
1. Is there any legal/licensing concern here? I am taking changes from
the bro-plugins version and pulling it into the Apache-hosted code. IANAL
2. Nick - are you OK with these changes?
Jon
On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
> Can someone on the PMC submit a ticket to INFRA? It looks like
> <https://www.apache.org/dev/infra-contact> committers aren't supposed to.
>
> Jon
>
> On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> I would be happy to try it again but I attempted to do that before with
> bro packages and it failed to be able to handle it. I also tried using
> branches of a repo with bro but that similarly failed (and was a pretty bad
> idea to start with).
>
> Jon
>
> On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
>
> We should be able to request just one alternate repo from INFRA, and put a
> top hierarchical level in it that doesn’t include a maven pom. As far as
> maven and clients are concerned, it
>
> just increases by 1 the path length to the root of the repo.
>
> On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> Once we agree on a repo location to host this, I would be happy to put
> together the package and update our environments to use bro-pkg to
> install
> the plugin. I have created METRON-813
> <https://issues.apache.org/jira/browse/METRON-813> to track this and
> changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348>
> to be
> a sub-task.
>
> Otto - the bro packages model doesn't allow colocation with anything
> else.
> That said, if we have two similar situations, and given the INFRA
> example
> <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to
> before
> was requesting 9 repos, perhaps we just request two repos. Would
> someone
> else mind putting that request in?
>
> Jon
>
> On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> Could we create a separate repo for more than on thing? like put … um
> let’s say
> a maven plugin and the bro plugin?
>
>
>
> On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
>
> I agree with everything that I've read.
>
> One of the guys from Bro had contacted me a while back, letting me know
> that the packaging mechanism in Bro was ready for public consumption. I
> just have not had cycles to do anything with it yet. They are not
> wanting
> to host any of the plugins.
>
> I thought the package mechanism requires that a package live within
> its own
> repo (which Casey confirmed). This put me in a bind on how to tackle
> this. I don't want to personally host the plugin in my own Github
> repo. I
> would prefer that we host it in a community repo; either Bro or Metron.
> Since Bro is moving away from hosting their own plugins, that leaves
> Metron.
>
> It would be great if we could create a separate repo for the plugin.
> That
> solves the challenge of using the packaging mechanism.
>
> We do need to reconcile what is in bro/bro-plugins and what is in
> Metron.
> There are some enhancements that I and others have made that never
> made it
> back into Metron. They never made it back, because the original plan
> was
> just to switch to using the plugin from bro/bro-plugins before the
> idea of
> a packaging mechanism hit Bro. Reconciling should be fairly easy to
> see by
> just doing a diff.
>
> It would be great if others want to take on any of that work. I would
> be
> glad to offer any support that you need. Thanks, Jon!
>
>
>
>
>
> On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Ok, great.
> >
> > I agree, I definitely want to hear from Nick on the topic. My team is
> > currently looking into enhancing the plugin as well to potentially
> allow
> > sending to multiple clusters, investigating some issues we see when
> our
> bro
> > cluster is under load, turn it into a package, etc.
> >
> > The work you just did was on our to do list as well so I'm very
> excited
> to
> > see it come through.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com>
> wrote:
> >
> > I *think* it's possible. People do ask for mirrors of directories
> from
> > time to time (see https://issues.apache.org/jira/browse/INFRA-7060).
> If
> > we
> > think this is a good idea, we can pose it to INFRA as a request. I'd
> love
> > to see us be able to use the bro packaging infrastructure and get
> more
> > visibility for the plugin.
> >
> > I'd be particularly interested in Nick's opinion on this, though.
> >
> > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> >
> > > You can version packages -
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > package.html#package-
> > > versioning
> > >
> > > I agree that having a separate repo provided by Apache would be
> optimal,
> > I
> > > just don't know the process for that or if it was even reasonable
> to
> > > suggest.
> > >
> > > Jon
> > >
> > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com>
> wrote:
> > >
> > > > Looking at the bro packages, it appears that bro is expecting
> things
> to
> > > be
> > > > its own git repository. I wonder if we could either request INFRA
> > > provide
> > > > another repo for the bro-kafka plugin and integrate it into
> metron as
> a
> > > git
> > > > submodule *or* if we could request INFRA to create a github
> mirror of
> > the
> > > > metron-sensors/bro-kafka-plugin directory. I'm not sure how
> viable
> > > either
> > > > of those options are, frankly.
> > > >
> > > > One thing that I didn't see is how do you specify a particular
> release
> > of
> > > > the plugin that you want to install? For us, we'd want to
> release the
> > > > plugin along with the product. I didn't quite see how you'd push
> > > releases
> > > > for bro plugins.
> > > >
> > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> cestella@gmail.com>
> > > wrote:
> > > >
> > > > > So, I do agree with the concern. Is there a way to host the
> package
> > > > > within Metron? I definitely would like to see the
> modifications at
> > > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd
> love for
> > us
> > > to
> > > > > host the plugin.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
>
> > > > > wrote:
> > > > >
> > > > >> Today I was taking a look at METRON-812
> > > > >> <https://issues.apache.org/jira/browse/METRON-812>, which
> made me
> > > > recall
> > > > >> some conversations from a while back regarding where the bro
> kafka
> > > > plugin
> > > > >> should ultimately live, and how to update it.
> > > > >>
> > > > >> Back in METRON-348 <https://issues.apache.org/
> > jira/browse/METRON-348>
> > > I
> > > > >> brought up the fact that some important changes
> > > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > > >> 5348da0a5043a8353b4a0a8>
> > > > >> were made to the externally hosted version of the kafka
> plugin,
> and
> > > were
> > > > >> never introduced to Metron's hosted version (i.e. the one we
> use
> > > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > >> in vagrant when bro is installed). The conversation went down
> the
> > > route
> > > > >> of
> > > > >> discussing whether or not the bro kafka plugin code should
> continue
> > to
> > > > >> live
> > > > >> in Metron in the first place. Now, with METRON-812, I see us
> > further
> > > > >> muddying the waters of where to go for the right plugin, as
> our
> > > version
> > > > is
> > > > >> still missing the public changes but adds some very important
> new
> > > > >> functionality.
> > > > >>
> > > > >> I'd like to bring up the idea of using bro's packages
> > > > >> <https://github.com/bro/packages> framework, released in
> late 2016
> > > > >> <
> http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > >> (additional
> > > > >> documentation here <
> > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > >> >),
> > > > >> as a potential place for this to be hosted/referenced. This
> is a
> > > simple
> > > > >> and supported method (funded by Mozilla
> > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > > >> e-support-first-awards-made/>)
> > > > >> to install and uninstall bro scripts, plugins, etc., and it
> also
> > > allows
> > > > us
> > > > >> to continue to have enough control over updates to the plugin
> so
> > that
> > > it
> > > > >> will not slow down Metron development by having it as a
> dependency
> > > > >> (resolving both of Casey's concerns noted here
> > > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > >> and I think this solution is supported by Nick's comments here
> > > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > >> as
> > > > >> well).
> > > > >>
> > > > >> The only thing I'm not sure about is where to host the plugin
> itself
> > -
> > > > my
> > > > >> first thought would be Nick's github <
> https://github.com/nickwallen
> > >,
> > > > as
> > > > >> he
> > > > >> really kicked off this effort, but maybe we can think of
> something
> > > > better.
> > > > >>
> > > > >> Is this approach of interest to anybody? It is extremely
> simple to
> > > put
> > > > >> together - I was able to throw one together
> > > > >> <https://github.com/bro/packages/blob/master/
> > jonzeolla/bro-pkg.index>
> > > > and
> > > > >> get it working with a fresh bro 2.5 install when attending the
> > brocon
> > > > talk
> > > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > >> bro-packagemanager>
> > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>,
> slides
> > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> > introduced
> > > > >> this
> > > > >> to me in the first place.
> > > > >>
> > > > >> Jon
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
>
>
> --
>
> Jon
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Can someone on the PMC submit a ticket to INFRA? It looks like
<https://www.apache.org/dev/infra-contact> committers aren't supposed to.
Jon
On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
> I would be happy to try it again but I attempted to do that before with
> bro packages and it failed to be able to handle it. I also tried using
> branches of a repo with bro but that similarly failed (and was a pretty bad
> idea to start with).
>
> Jon
>
> On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
>
> We should be able to request just one alternate repo from INFRA, and put a
> top hierarchical level in it that doesn’t include a maven pom. As far as
> maven and clients are concerned, it
>
> just increases by 1 the path length to the root of the repo.
>
> On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> Once we agree on a repo location to host this, I would be happy to put
> together the package and update our environments to use bro-pkg to
> install
> the plugin. I have created METRON-813
> <https://issues.apache.org/jira/browse/METRON-813> to track this and
> changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348>
> to be
> a sub-task.
>
> Otto - the bro packages model doesn't allow colocation with anything
> else.
> That said, if we have two similar situations, and given the INFRA
> example
> <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to
> before
> was requesting 9 repos, perhaps we just request two repos. Would
> someone
> else mind putting that request in?
>
> Jon
>
> On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> Could we create a separate repo for more than on thing? like put … um
> let’s say
> a maven plugin and the bro plugin?
>
>
>
> On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
>
> I agree with everything that I've read.
>
> One of the guys from Bro had contacted me a while back, letting me know
> that the packaging mechanism in Bro was ready for public consumption. I
> just have not had cycles to do anything with it yet. They are not
> wanting
> to host any of the plugins.
>
> I thought the package mechanism requires that a package live within
> its own
> repo (which Casey confirmed). This put me in a bind on how to tackle
> this. I don't want to personally host the plugin in my own Github
> repo. I
> would prefer that we host it in a community repo; either Bro or Metron.
> Since Bro is moving away from hosting their own plugins, that leaves
> Metron.
>
> It would be great if we could create a separate repo for the plugin.
> That
> solves the challenge of using the packaging mechanism.
>
> We do need to reconcile what is in bro/bro-plugins and what is in
> Metron.
> There are some enhancements that I and others have made that never
> made it
> back into Metron. They never made it back, because the original plan
> was
> just to switch to using the plugin from bro/bro-plugins before the
> idea of
> a packaging mechanism hit Bro. Reconciling should be fairly easy to
> see by
> just doing a diff.
>
> It would be great if others want to take on any of that work. I would
> be
> glad to offer any support that you need. Thanks, Jon!
>
>
>
>
>
> On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Ok, great.
> >
> > I agree, I definitely want to hear from Nick on the topic. My team is
> > currently looking into enhancing the plugin as well to potentially
> allow
> > sending to multiple clusters, investigating some issues we see when
> our
> bro
> > cluster is under load, turn it into a package, etc.
> >
> > The work you just did was on our to do list as well so I'm very
> excited
> to
> > see it come through.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com>
> wrote:
> >
> > I *think* it's possible. People do ask for mirrors of directories
> from
> > time to time (see https://issues.apache.org/jira/browse/INFRA-7060).
> If
> > we
> > think this is a good idea, we can pose it to INFRA as a request. I'd
> love
> > to see us be able to use the bro packaging infrastructure and get
> more
> > visibility for the plugin.
> >
> > I'd be particularly interested in Nick's opinion on this, though.
> >
> > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> >
> > > You can version packages -
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > package.html#package-
> > > versioning
> > >
> > > I agree that having a separate repo provided by Apache would be
> optimal,
> > I
> > > just don't know the process for that or if it was even reasonable
> to
> > > suggest.
> > >
> > > Jon
> > >
> > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com>
> wrote:
> > >
> > > > Looking at the bro packages, it appears that bro is expecting
> things
> to
> > > be
> > > > its own git repository. I wonder if we could either request INFRA
> > > provide
> > > > another repo for the bro-kafka plugin and integrate it into
> metron as
> a
> > > git
> > > > submodule *or* if we could request INFRA to create a github
> mirror of
> > the
> > > > metron-sensors/bro-kafka-plugin directory. I'm not sure how
> viable
> > > either
> > > > of those options are, frankly.
> > > >
> > > > One thing that I didn't see is how do you specify a particular
> release
> > of
> > > > the plugin that you want to install? For us, we'd want to
> release the
> > > > plugin along with the product. I didn't quite see how you'd push
> > > releases
> > > > for bro plugins.
> > > >
> > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> cestella@gmail.com>
> > > wrote:
> > > >
> > > > > So, I do agree with the concern. Is there a way to host the
> package
> > > > > within Metron? I definitely would like to see the
> modifications at
> > > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd
> love for
> > us
> > > to
> > > > > host the plugin.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
>
> > > > > wrote:
> > > > >
> > > > >> Today I was taking a look at METRON-812
> > > > >> <https://issues.apache.org/jira/browse/METRON-812>, which
> made me
> > > > recall
> > > > >> some conversations from a while back regarding where the bro
> kafka
> > > > plugin
> > > > >> should ultimately live, and how to update it.
> > > > >>
> > > > >> Back in METRON-348 <https://issues.apache.org/
> > jira/browse/METRON-348>
> > > I
> > > > >> brought up the fact that some important changes
> > > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > > >> 5348da0a5043a8353b4a0a8>
> > > > >> were made to the externally hosted version of the kafka
> plugin,
> and
> > > were
> > > > >> never introduced to Metron's hosted version (i.e. the one we
> use
> > > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > >> in vagrant when bro is installed). The conversation went down
> the
> > > route
> > > > >> of
> > > > >> discussing whether or not the bro kafka plugin code should
> continue
> > to
> > > > >> live
> > > > >> in Metron in the first place. Now, with METRON-812, I see us
> > further
> > > > >> muddying the waters of where to go for the right plugin, as
> our
> > > version
> > > > is
> > > > >> still missing the public changes but adds some very important
> new
> > > > >> functionality.
> > > > >>
> > > > >> I'd like to bring up the idea of using bro's packages
> > > > >> <https://github.com/bro/packages> framework, released in
> late 2016
> > > > >> <
> http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > >> (additional
> > > > >> documentation here <
> > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > >> >),
> > > > >> as a potential place for this to be hosted/referenced. This
> is a
> > > simple
> > > > >> and supported method (funded by Mozilla
> > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > > >> e-support-first-awards-made/>)
> > > > >> to install and uninstall bro scripts, plugins, etc., and it
> also
> > > allows
> > > > us
> > > > >> to continue to have enough control over updates to the plugin
> so
> > that
> > > it
> > > > >> will not slow down Metron development by having it as a
> dependency
> > > > >> (resolving both of Casey's concerns noted here
> > > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > >> and I think this solution is supported by Nick's comments here
> > > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > >> as
> > > > >> well).
> > > > >>
> > > > >> The only thing I'm not sure about is where to host the plugin
> itself
> > -
> > > > my
> > > > >> first thought would be Nick's github <
> https://github.com/nickwallen
> > >,
> > > > as
> > > > >> he
> > > > >> really kicked off this effort, but maybe we can think of
> something
> > > > better.
> > > > >>
> > > > >> Is this approach of interest to anybody? It is extremely
> simple to
> > > put
> > > > >> together - I was able to throw one together
> > > > >> <https://github.com/bro/packages/blob/master/
> > jonzeolla/bro-pkg.index>
> > > > and
> > > > >> get it working with a fresh bro 2.5 install when attending the
> > brocon
> > > > talk
> > > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > >> bro-packagemanager>
> > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>,
> slides
> > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> > introduced
> > > > >> this
> > > > >> to me in the first place.
> > > > >>
> > > > >> Jon
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
>
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I would be happy to try it again but I attempted to do that before with bro
packages and it failed to be able to handle it. I also tried using
branches of a repo with bro but that similarly failed (and was a pretty bad
idea to start with).
Jon
On Fri, Mar 31, 2017, 3:24 PM Matt Foley <ma...@apache.org> wrote:
> We should be able to request just one alternate repo from INFRA, and put a
> top hierarchical level in it that doesn’t include a maven pom. As far as
> maven and clients are concerned, it
>
> just increases by 1 the path length to the root of the repo.
>
> On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
>
> Once we agree on a repo location to host this, I would be happy to put
> together the package and update our environments to use bro-pkg to
> install
> the plugin. I have created METRON-813
> <https://issues.apache.org/jira/browse/METRON-813> to track this and
> changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348>
> to be
> a sub-task.
>
> Otto - the bro packages model doesn't allow colocation with anything
> else.
> That said, if we have two similar situations, and given the INFRA
> example
> <https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to
> before
> was requesting 9 repos, perhaps we just request two repos. Would
> someone
> else mind putting that request in?
>
> Jon
>
> On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> Could we create a separate repo for more than on thing? like put … um
> let’s say
> a maven plugin and the bro plugin?
>
>
>
> On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
>
> I agree with everything that I've read.
>
> One of the guys from Bro had contacted me a while back, letting me know
> that the packaging mechanism in Bro was ready for public consumption. I
> just have not had cycles to do anything with it yet. They are not
> wanting
> to host any of the plugins.
>
> I thought the package mechanism requires that a package live within
> its own
> repo (which Casey confirmed). This put me in a bind on how to tackle
> this. I don't want to personally host the plugin in my own Github
> repo. I
> would prefer that we host it in a community repo; either Bro or Metron.
> Since Bro is moving away from hosting their own plugins, that leaves
> Metron.
>
> It would be great if we could create a separate repo for the plugin.
> That
> solves the challenge of using the packaging mechanism.
>
> We do need to reconcile what is in bro/bro-plugins and what is in
> Metron.
> There are some enhancements that I and others have made that never
> made it
> back into Metron. They never made it back, because the original plan
> was
> just to switch to using the plugin from bro/bro-plugins before the
> idea of
> a packaging mechanism hit Bro. Reconciling should be fairly easy to
> see by
> just doing a diff.
>
> It would be great if others want to take on any of that work. I would
> be
> glad to offer any support that you need. Thanks, Jon!
>
>
>
>
>
> On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Ok, great.
> >
> > I agree, I definitely want to hear from Nick on the topic. My team is
> > currently looking into enhancing the plugin as well to potentially
> allow
> > sending to multiple clusters, investigating some issues we see when
> our
> bro
> > cluster is under load, turn it into a package, etc.
> >
> > The work you just did was on our to do list as well so I'm very
> excited
> to
> > see it come through.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com>
> wrote:
> >
> > I *think* it's possible. People do ask for mirrors of directories
> from
> > time to time (see https://issues.apache.org/jira/browse/INFRA-7060).
> If
> > we
> > think this is a good idea, we can pose it to INFRA as a request. I'd
> love
> > to see us be able to use the bro packaging infrastructure and get
> more
> > visibility for the plugin.
> >
> > I'd be particularly interested in Nick's opinion on this, though.
> >
> > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> >
> > > You can version packages -
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > package.html#package-
> > > versioning
> > >
> > > I agree that having a separate repo provided by Apache would be
> optimal,
> > I
> > > just don't know the process for that or if it was even reasonable
> to
> > > suggest.
> > >
> > > Jon
> > >
> > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com>
> wrote:
> > >
> > > > Looking at the bro packages, it appears that bro is expecting
> things
> to
> > > be
> > > > its own git repository. I wonder if we could either request INFRA
> > > provide
> > > > another repo for the bro-kafka plugin and integrate it into
> metron as
> a
> > > git
> > > > submodule *or* if we could request INFRA to create a github
> mirror of
> > the
> > > > metron-sensors/bro-kafka-plugin directory. I'm not sure how
> viable
> > > either
> > > > of those options are, frankly.
> > > >
> > > > One thing that I didn't see is how do you specify a particular
> release
> > of
> > > > the plugin that you want to install? For us, we'd want to
> release the
> > > > plugin along with the product. I didn't quite see how you'd push
> > > releases
> > > > for bro plugins.
> > > >
> > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <
> cestella@gmail.com>
> > > wrote:
> > > >
> > > > > So, I do agree with the concern. Is there a way to host the
> package
> > > > > within Metron? I definitely would like to see the
> modifications at
> > > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd
> love for
> > us
> > > to
> > > > > host the plugin.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
>
> > > > > wrote:
> > > > >
> > > > >> Today I was taking a look at METRON-812
> > > > >> <https://issues.apache.org/jira/browse/METRON-812>, which
> made me
> > > > recall
> > > > >> some conversations from a while back regarding where the bro
> kafka
> > > > plugin
> > > > >> should ultimately live, and how to update it.
> > > > >>
> > > > >> Back in METRON-348 <https://issues.apache.org/
> > jira/browse/METRON-348>
> > > I
> > > > >> brought up the fact that some important changes
> > > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > > >> 5348da0a5043a8353b4a0a8>
> > > > >> were made to the externally hosted version of the kafka
> plugin,
> and
> > > were
> > > > >> never introduced to Metron's hosted version (i.e. the one we
> use
> > > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > > >> in vagrant when bro is installed). The conversation went down
> the
> > > route
> > > > >> of
> > > > >> discussing whether or not the bro kafka plugin code should
> continue
> > to
> > > > >> live
> > > > >> in Metron in the first place. Now, with METRON-812, I see us
> > further
> > > > >> muddying the waters of where to go for the right plugin, as
> our
> > > version
> > > > is
> > > > >> still missing the public changes but adds some very important
> new
> > > > >> functionality.
> > > > >>
> > > > >> I'd like to bring up the idea of using bro's packages
> > > > >> <https://github.com/bro/packages> framework, released in
> late 2016
> > > > >> <
> http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > > >> (additional
> > > > >> documentation here <
> > > > http://bro-package-manager.readthedocs.io/en/stable/
> > > > >> >),
> > > > >> as a potential place for this to be hosted/referenced. This
> is a
> > > simple
> > > > >> and supported method (funded by Mozilla
> > > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > > >> e-support-first-awards-made/>)
> > > > >> to install and uninstall bro scripts, plugins, etc., and it
> also
> > > allows
> > > > us
> > > > >> to continue to have enough control over updates to the plugin
> so
> > that
> > > it
> > > > >> will not slow down Metron development by having it as a
> dependency
> > > > >> (resolving both of Casey's concerns noted here
> > > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > > >> and I think this solution is supported by Nick's comments here
> > > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > > >> as
> > > > >> well).
> > > > >>
> > > > >> The only thing I'm not sure about is where to host the plugin
> itself
> > -
> > > > my
> > > > >> first thought would be Nick's github <
> https://github.com/nickwallen
> > >,
> > > > as
> > > > >> he
> > > > >> really kicked off this effort, but maybe we can think of
> something
> > > > better.
> > > > >>
> > > > >> Is this approach of interest to anybody? It is extremely
> simple to
> > > put
> > > > >> together - I was able to throw one together
> > > > >> <https://github.com/bro/packages/blob/master/
> > jonzeolla/bro-pkg.index>
> > > > and
> > > > >> get it working with a fresh bro 2.5 install when attending the
> > brocon
> > > > talk
> > > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > > >> bro-packagemanager>
> > > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>,
> slides
> > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> > introduced
> > > > >> this
> > > > >> to me in the first place.
> > > > >>
> > > > >> Jon
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
>
>
> --
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Matt Foley <ma...@apache.org>.
We should be able to request just one alternate repo from INFRA, and put a top hierarchical level in it that doesn’t include a maven pom. As far as maven and clients are concerned, it
just increases by 1 the path length to the root of the repo.
On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:
Once we agree on a repo location to host this, I would be happy to put
together the package and update our environments to use bro-pkg to install
the plugin. I have created METRON-813
<https://issues.apache.org/jira/browse/METRON-813> to track this and
changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348> to be
a sub-task.
Otto - the bro packages model doesn't allow colocation with anything else.
That said, if we have two similar situations, and given the INFRA example
<https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to before
was requesting 9 repos, perhaps we just request two repos. Would someone
else mind putting that request in?
Jon
On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <ot...@gmail.com>
wrote:
Could we create a separate repo for more than on thing? like put … um
let’s say
a maven plugin and the bro plugin?
On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
I agree with everything that I've read.
One of the guys from Bro had contacted me a while back, letting me know
that the packaging mechanism in Bro was ready for public consumption. I
just have not had cycles to do anything with it yet. They are not wanting
to host any of the plugins.
I thought the package mechanism requires that a package live within its own
repo (which Casey confirmed). This put me in a bind on how to tackle
this. I don't want to personally host the plugin in my own Github repo. I
would prefer that we host it in a community repo; either Bro or Metron.
Since Bro is moving away from hosting their own plugins, that leaves
Metron.
It would be great if we could create a separate repo for the plugin. That
solves the challenge of using the packaging mechanism.
We do need to reconcile what is in bro/bro-plugins and what is in Metron.
There are some enhancements that I and others have made that never made it
back into Metron. They never made it back, because the original plan was
just to switch to using the plugin from bro/bro-plugins before the idea of
a packaging mechanism hit Bro. Reconciling should be fairly easy to see by
just doing a diff.
It would be great if others want to take on any of that work. I would be
glad to offer any support that you need. Thanks, Jon!
On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
> Ok, great.
>
> I agree, I definitely want to hear from Nick on the topic. My team is
> currently looking into enhancing the plugin as well to potentially allow
> sending to multiple clusters, investigating some issues we see when our
bro
> cluster is under load, turn it into a package, etc.
>
> The work you just did was on our to do list as well so I'm very excited
to
> see it come through.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com> wrote:
>
> I *think* it's possible. People do ask for mirrors of directories from
> time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If
> we
> think this is a good idea, we can pose it to INFRA as a request. I'd love
> to see us be able to use the bro packaging infrastructure and get more
> visibility for the plugin.
>
> I'd be particularly interested in Nick's opinion on this, though.
>
> On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > You can version packages -
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > versioning
> >
> > I agree that having a separate repo provided by Apache would be
optimal,
> I
> > just don't know the process for that or if it was even reasonable to
> > suggest.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > Looking at the bro packages, it appears that bro is expecting things
to
> > be
> > > its own git repository. I wonder if we could either request INFRA
> > provide
> > > another repo for the bro-kafka plugin and integrate it into metron as
a
> > git
> > > submodule *or* if we could request INFRA to create a github mirror of
> the
> > > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> > either
> > > of those options are, frankly.
> > >
> > > One thing that I didn't see is how do you specify a particular
release
> of
> > > the plugin that you want to install? For us, we'd want to release the
> > > plugin along with the product. I didn't quite see how you'd push
> > releases
> > > for bro plugins.
> > >
> > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> > >
> > > > So, I do agree with the concern. Is there a way to host the package
> > > > within Metron? I definitely would like to see the modifications at
> > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for
> us
> > to
> > > > host the plugin.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > > wrote:
> > > >
> > > >> Today I was taking a look at METRON-812
> > > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > > recall
> > > >> some conversations from a while back regarding where the bro kafka
> > > plugin
> > > >> should ultimately live, and how to update it.
> > > >>
> > > >> Back in METRON-348 <https://issues.apache.org/
> jira/browse/METRON-348>
> > I
> > > >> brought up the fact that some important changes
> > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > >> 5348da0a5043a8353b4a0a8>
> > > >> were made to the externally hosted version of the kafka plugin,
and
> > were
> > > >> never introduced to Metron's hosted version (i.e. the one we use
> > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > >> in vagrant when bro is installed). The conversation went down the
> > route
> > > >> of
> > > >> discussing whether or not the bro kafka plugin code should
continue
> to
> > > >> live
> > > >> in Metron in the first place. Now, with METRON-812, I see us
> further
> > > >> muddying the waters of where to go for the right plugin, as our
> > version
> > > is
> > > >> still missing the public changes but adds some very important new
> > > >> functionality.
> > > >>
> > > >> I'd like to bring up the idea of using bro's packages
> > > >> <https://github.com/bro/packages> framework, released in late 2016
> > > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > >> (additional
> > > >> documentation here <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > > >> >),
> > > >> as a potential place for this to be hosted/referenced. This is a
> > simple
> > > >> and supported method (funded by Mozilla
> > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > >> e-support-first-awards-made/>)
> > > >> to install and uninstall bro scripts, plugins, etc., and it also
> > allows
> > > us
> > > >> to continue to have enough control over updates to the plugin so
> that
> > it
> > > >> will not slow down Metron development by having it as a dependency
> > > >> (resolving both of Casey's concerns noted here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > >> and I think this solution is supported by Nick's comments here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > >> as
> > > >> well).
> > > >>
> > > >> The only thing I'm not sure about is where to host the plugin
itself
> -
> > > my
> > > >> first thought would be Nick's github <https://github.com/nickwallen
> >,
> > > as
> > > >> he
> > > >> really kicked off this effort, but maybe we can think of something
> > > better.
> > > >>
> > > >> Is this approach of interest to anybody? It is extremely simple to
> > put
> > > >> together - I was able to throw one together
> > > >> <https://github.com/bro/packages/blob/master/
> jonzeolla/bro-pkg.index>
> > > and
> > > >> get it working with a fresh bro 2.5 install when attending the
> brocon
> > > talk
> > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > >> bro-packagemanager>
> > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> introduced
> > > >> this
> > > >> to me in the first place.
> > > >>
> > > >> Jon
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Otto Fowler <ot...@gmail.com>.
I don’t want to put the request for the repo for the plugin until it stands
review.
On March 31, 2017 at 13:30:45, Zeolla@GMail.com (zeolla@gmail.com) wrote:
Once we agree on a repo location to host this, I would be happy to put
together the package and update our environments to use bro-pkg to install
the plugin. I have created METRON-813
<https://issues.apache.org/jira/browse/METRON-813> to track this and
changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348> to be
a sub-task.
Otto - the bro packages model doesn't allow colocation with anything else.
That said, if we have two similar situations, and given the INFRA example
<https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to before
was requesting 9 repos, perhaps we just request two repos. Would someone
else mind putting that request in?
Jon
On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <ot...@gmail.com>
wrote:
Could we create a separate repo for more than on thing? like put … um
let’s say
a maven plugin and the bro plugin?
On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
I agree with everything that I've read.
One of the guys from Bro had contacted me a while back, letting me know
that the packaging mechanism in Bro was ready for public consumption. I
just have not had cycles to do anything with it yet. They are not wanting
to host any of the plugins.
I thought the package mechanism requires that a package live within its own
repo (which Casey confirmed). This put me in a bind on how to tackle
this. I don't want to personally host the plugin in my own Github repo. I
would prefer that we host it in a community repo; either Bro or Metron.
Since Bro is moving away from hosting their own plugins, that leaves
Metron.
It would be great if we could create a separate repo for the plugin. That
solves the challenge of using the packaging mechanism.
We do need to reconcile what is in bro/bro-plugins and what is in Metron.
There are some enhancements that I and others have made that never made it
back into Metron. They never made it back, because the original plan was
just to switch to using the plugin from bro/bro-plugins before the idea of
a packaging mechanism hit Bro. Reconciling should be fairly easy to see by
just doing a diff.
It would be great if others want to take on any of that work. I would be
glad to offer any support that you need. Thanks, Jon!
On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
> Ok, great.
>
> I agree, I definitely want to hear from Nick on the topic. My team is
> currently looking into enhancing the plugin as well to potentially allow
> sending to multiple clusters, investigating some issues we see when our
bro
> cluster is under load, turn it into a package, etc.
>
> The work you just did was on our to do list as well so I'm very excited
to
> see it come through.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com> wrote:
>
> I *think* it's possible. People do ask for mirrors of directories from
> time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If
> we
> think this is a good idea, we can pose it to INFRA as a request. I'd love
> to see us be able to use the bro packaging infrastructure and get more
> visibility for the plugin.
>
> I'd be particularly interested in Nick's opinion on this, though.
>
> On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > You can version packages -
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > versioning
> >
> > I agree that having a separate repo provided by Apache would be
optimal,
> I
> > just don't know the process for that or if it was even reasonable to
> > suggest.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > Looking at the bro packages, it appears that bro is expecting things
to
> > be
> > > its own git repository. I wonder if we could either request INFRA
> > provide
> > > another repo for the bro-kafka plugin and integrate it into metron as
a
> > git
> > > submodule *or* if we could request INFRA to create a github mirror of
> the
> > > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> > either
> > > of those options are, frankly.
> > >
> > > One thing that I didn't see is how do you specify a particular
release
> of
> > > the plugin that you want to install? For us, we'd want to release the
> > > plugin along with the product. I didn't quite see how you'd push
> > releases
> > > for bro plugins.
> > >
> > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> > >
> > > > So, I do agree with the concern. Is there a way to host the package
> > > > within Metron? I definitely would like to see the modifications at
> > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for
> us
> > to
> > > > host the plugin.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > > wrote:
> > > >
> > > >> Today I was taking a look at METRON-812
> > > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > > recall
> > > >> some conversations from a while back regarding where the bro kafka
> > > plugin
> > > >> should ultimately live, and how to update it.
> > > >>
> > > >> Back in METRON-348 <https://issues.apache.org/
> jira/browse/METRON-348>
> > I
> > > >> brought up the fact that some important changes
> > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > >> 5348da0a5043a8353b4a0a8>
> > > >> were made to the externally hosted version of the kafka plugin,
and
> > were
> > > >> never introduced to Metron's hosted version (i.e. the one we use
> > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > >> in vagrant when bro is installed). The conversation went down the
> > route
> > > >> of
> > > >> discussing whether or not the bro kafka plugin code should
continue
> to
> > > >> live
> > > >> in Metron in the first place. Now, with METRON-812, I see us
> further
> > > >> muddying the waters of where to go for the right plugin, as our
> > version
> > > is
> > > >> still missing the public changes but adds some very important new
> > > >> functionality.
> > > >>
> > > >> I'd like to bring up the idea of using bro's packages
> > > >> <https://github.com/bro/packages> framework, released in late 2016
> > > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > >> (additional
> > > >> documentation here <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > > >> >),
> > > >> as a potential place for this to be hosted/referenced. This is a
> > simple
> > > >> and supported method (funded by Mozilla
> > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > >> e-support-first-awards-made/>)
> > > >> to install and uninstall bro scripts, plugins, etc., and it also
> > allows
> > > us
> > > >> to continue to have enough control over updates to the plugin so
> that
> > it
> > > >> will not slow down Metron development by having it as a dependency
> > > >> (resolving both of Casey's concerns noted here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > >> and I think this solution is supported by Nick's comments here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > >> as
> > > >> well).
> > > >>
> > > >> The only thing I'm not sure about is where to host the plugin
itself
> -
> > > my
> > > >> first thought would be Nick's github <https://github.com/nickwallen
> >,
> > > as
> > > >> he
> > > >> really kicked off this effort, but maybe we can think of something
> > > better.
> > > >>
> > > >> Is this approach of interest to anybody? It is extremely simple to
> > put
> > > >> together - I was able to throw one together
> > > >> <https://github.com/bro/packages/blob/master/
> jonzeolla/bro-pkg.index>
> > > and
> > > >> get it working with a fresh bro 2.5 install when attending the
> brocon
> > > talk
> > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > >> bro-packagemanager>
> > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> introduced
> > > >> this
> > > >> to me in the first place.
> > > >>
> > > >> Jon
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Once we agree on a repo location to host this, I would be happy to put
together the package and update our environments to use bro-pkg to install
the plugin. I have created METRON-813
<https://issues.apache.org/jira/browse/METRON-813> to track this and
changed METRON-348 <https://issues.apache.org/jira/browse/METRON-348> to be
a sub-task.
Otto - the bro packages model doesn't allow colocation with anything else.
That said, if we have two similar situations, and given the INFRA example
<https://issues.apache.org/jira/browse/INFRA-7060> Casey linked to before
was requesting 9 repos, perhaps we just request two repos. Would someone
else mind putting that request in?
Jon
On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <ot...@gmail.com>
wrote:
Could we create a separate repo for more than on thing? like put … um
let’s say
a maven plugin and the bro plugin?
On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
I agree with everything that I've read.
One of the guys from Bro had contacted me a while back, letting me know
that the packaging mechanism in Bro was ready for public consumption. I
just have not had cycles to do anything with it yet. They are not wanting
to host any of the plugins.
I thought the package mechanism requires that a package live within its own
repo (which Casey confirmed). This put me in a bind on how to tackle
this. I don't want to personally host the plugin in my own Github repo. I
would prefer that we host it in a community repo; either Bro or Metron.
Since Bro is moving away from hosting their own plugins, that leaves
Metron.
It would be great if we could create a separate repo for the plugin. That
solves the challenge of using the packaging mechanism.
We do need to reconcile what is in bro/bro-plugins and what is in Metron.
There are some enhancements that I and others have made that never made it
back into Metron. They never made it back, because the original plan was
just to switch to using the plugin from bro/bro-plugins before the idea of
a packaging mechanism hit Bro. Reconciling should be fairly easy to see by
just doing a diff.
It would be great if others want to take on any of that work. I would be
glad to offer any support that you need. Thanks, Jon!
On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
> Ok, great.
>
> I agree, I definitely want to hear from Nick on the topic. My team is
> currently looking into enhancing the plugin as well to potentially allow
> sending to multiple clusters, investigating some issues we see when our
bro
> cluster is under load, turn it into a package, etc.
>
> The work you just did was on our to do list as well so I'm very excited
to
> see it come through.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com> wrote:
>
> I *think* it's possible. People do ask for mirrors of directories from
> time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If
> we
> think this is a good idea, we can pose it to INFRA as a request. I'd love
> to see us be able to use the bro packaging infrastructure and get more
> visibility for the plugin.
>
> I'd be particularly interested in Nick's opinion on this, though.
>
> On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > You can version packages -
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > versioning
> >
> > I agree that having a separate repo provided by Apache would be
optimal,
> I
> > just don't know the process for that or if it was even reasonable to
> > suggest.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > Looking at the bro packages, it appears that bro is expecting things
to
> > be
> > > its own git repository. I wonder if we could either request INFRA
> > provide
> > > another repo for the bro-kafka plugin and integrate it into metron as
a
> > git
> > > submodule *or* if we could request INFRA to create a github mirror of
> the
> > > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> > either
> > > of those options are, frankly.
> > >
> > > One thing that I didn't see is how do you specify a particular
release
> of
> > > the plugin that you want to install? For us, we'd want to release the
> > > plugin along with the product. I didn't quite see how you'd push
> > releases
> > > for bro plugins.
> > >
> > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> > >
> > > > So, I do agree with the concern. Is there a way to host the package
> > > > within Metron? I definitely would like to see the modifications at
> > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for
> us
> > to
> > > > host the plugin.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > > wrote:
> > > >
> > > >> Today I was taking a look at METRON-812
> > > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > > recall
> > > >> some conversations from a while back regarding where the bro kafka
> > > plugin
> > > >> should ultimately live, and how to update it.
> > > >>
> > > >> Back in METRON-348 <https://issues.apache.org/
> jira/browse/METRON-348>
> > I
> > > >> brought up the fact that some important changes
> > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > >> 5348da0a5043a8353b4a0a8>
> > > >> were made to the externally hosted version of the kafka plugin,
and
> > were
> > > >> never introduced to Metron's hosted version (i.e. the one we use
> > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > >> in vagrant when bro is installed). The conversation went down the
> > route
> > > >> of
> > > >> discussing whether or not the bro kafka plugin code should
continue
> to
> > > >> live
> > > >> in Metron in the first place. Now, with METRON-812, I see us
> further
> > > >> muddying the waters of where to go for the right plugin, as our
> > version
> > > is
> > > >> still missing the public changes but adds some very important new
> > > >> functionality.
> > > >>
> > > >> I'd like to bring up the idea of using bro's packages
> > > >> <https://github.com/bro/packages> framework, released in late 2016
> > > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > >> (additional
> > > >> documentation here <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > > >> >),
> > > >> as a potential place for this to be hosted/referenced. This is a
> > simple
> > > >> and supported method (funded by Mozilla
> > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > >> e-support-first-awards-made/>)
> > > >> to install and uninstall bro scripts, plugins, etc., and it also
> > allows
> > > us
> > > >> to continue to have enough control over updates to the plugin so
> that
> > it
> > > >> will not slow down Metron development by having it as a dependency
> > > >> (resolving both of Casey's concerns noted here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > >> and I think this solution is supported by Nick's comments here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > >> as
> > > >> well).
> > > >>
> > > >> The only thing I'm not sure about is where to host the plugin
itself
> -
> > > my
> > > >> first thought would be Nick's github <https://github.com/nickwallen
> >,
> > > as
> > > >> he
> > > >> really kicked off this effort, but maybe we can think of something
> > > better.
> > > >>
> > > >> Is this approach of interest to anybody? It is extremely simple to
> > put
> > > >> together - I was able to throw one together
> > > >> <https://github.com/bro/packages/blob/master/
> jonzeolla/bro-pkg.index>
> > > and
> > > >> get it working with a fresh bro 2.5 install when attending the
> brocon
> > > talk
> > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > >> bro-packagemanager>
> > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> introduced
> > > >> this
> > > >> to me in the first place.
> > > >>
> > > >> Jon
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Otto Fowler <ot...@gmail.com>.
Could we create a separate repo for more than on thing? like put … um
let’s say
a maven plugin and the bro plugin?
On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org) wrote:
I agree with everything that I've read.
One of the guys from Bro had contacted me a while back, letting me know
that the packaging mechanism in Bro was ready for public consumption. I
just have not had cycles to do anything with it yet. They are not wanting
to host any of the plugins.
I thought the package mechanism requires that a package live within its own
repo (which Casey confirmed). This put me in a bind on how to tackle
this. I don't want to personally host the plugin in my own Github repo. I
would prefer that we host it in a community repo; either Bro or Metron.
Since Bro is moving away from hosting their own plugins, that leaves
Metron.
It would be great if we could create a separate repo for the plugin. That
solves the challenge of using the packaging mechanism.
We do need to reconcile what is in bro/bro-plugins and what is in Metron.
There are some enhancements that I and others have made that never made it
back into Metron. They never made it back, because the original plan was
just to switch to using the plugin from bro/bro-plugins before the idea of
a packaging mechanism hit Bro. Reconciling should be fairly easy to see by
just doing a diff.
It would be great if others want to take on any of that work. I would be
glad to offer any support that you need. Thanks, Jon!
On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
> Ok, great.
>
> I agree, I definitely want to hear from Nick on the topic. My team is
> currently looking into enhancing the plugin as well to potentially allow
> sending to multiple clusters, investigating some issues we see when our
bro
> cluster is under load, turn it into a package, etc.
>
> The work you just did was on our to do list as well so I'm very excited
to
> see it come through.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com> wrote:
>
> I *think* it's possible. People do ask for mirrors of directories from
> time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If
> we
> think this is a good idea, we can pose it to INFRA as a request. I'd love
> to see us be able to use the bro packaging infrastructure and get more
> visibility for the plugin.
>
> I'd be particularly interested in Nick's opinion on this, though.
>
> On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > You can version packages -
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > versioning
> >
> > I agree that having a separate repo provided by Apache would be
optimal,
> I
> > just don't know the process for that or if it was even reasonable to
> > suggest.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > Looking at the bro packages, it appears that bro is expecting things
to
> > be
> > > its own git repository. I wonder if we could either request INFRA
> > provide
> > > another repo for the bro-kafka plugin and integrate it into metron as
a
> > git
> > > submodule *or* if we could request INFRA to create a github mirror of
> the
> > > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> > either
> > > of those options are, frankly.
> > >
> > > One thing that I didn't see is how do you specify a particular
release
> of
> > > the plugin that you want to install? For us, we'd want to release the
> > > plugin along with the product. I didn't quite see how you'd push
> > releases
> > > for bro plugins.
> > >
> > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> > >
> > > > So, I do agree with the concern. Is there a way to host the package
> > > > within Metron? I definitely would like to see the modifications at
> > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for
> us
> > to
> > > > host the plugin.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > > wrote:
> > > >
> > > >> Today I was taking a look at METRON-812
> > > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > > recall
> > > >> some conversations from a while back regarding where the bro kafka
> > > plugin
> > > >> should ultimately live, and how to update it.
> > > >>
> > > >> Back in METRON-348 <https://issues.apache.org/
> jira/browse/METRON-348>
> > I
> > > >> brought up the fact that some important changes
> > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > >> 5348da0a5043a8353b4a0a8>
> > > >> were made to the externally hosted version of the kafka plugin,
and
> > were
> > > >> never introduced to Metron's hosted version (i.e. the one we use
> > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > >> in vagrant when bro is installed). The conversation went down the
> > route
> > > >> of
> > > >> discussing whether or not the bro kafka plugin code should
continue
> to
> > > >> live
> > > >> in Metron in the first place. Now, with METRON-812, I see us
> further
> > > >> muddying the waters of where to go for the right plugin, as our
> > version
> > > is
> > > >> still missing the public changes but adds some very important new
> > > >> functionality.
> > > >>
> > > >> I'd like to bring up the idea of using bro's packages
> > > >> <https://github.com/bro/packages> framework, released in late 2016
> > > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > >> (additional
> > > >> documentation here <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > > >> >),
> > > >> as a potential place for this to be hosted/referenced. This is a
> > simple
> > > >> and supported method (funded by Mozilla
> > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > >> e-support-first-awards-made/>)
> > > >> to install and uninstall bro scripts, plugins, etc., and it also
> > allows
> > > us
> > > >> to continue to have enough control over updates to the plugin so
> that
> > it
> > > >> will not slow down Metron development by having it as a dependency
> > > >> (resolving both of Casey's concerns noted here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > >> and I think this solution is supported by Nick's comments here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > >> as
> > > >> well).
> > > >>
> > > >> The only thing I'm not sure about is where to host the plugin
itself
> -
> > > my
> > > >> first thought would be Nick's github <https://github.com/nickwallen
> >,
> > > as
> > > >> he
> > > >> really kicked off this effort, but maybe we can think of something
> > > better.
> > > >>
> > > >> Is this approach of interest to anybody? It is extremely simple to
> > put
> > > >> together - I was able to throw one together
> > > >> <https://github.com/bro/packages/blob/master/
> jonzeolla/bro-pkg.index>
> > > and
> > > >> get it working with a fresh bro 2.5 install when attending the
> brocon
> > > talk
> > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > >> bro-packagemanager>
> > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> introduced
> > > >> this
> > > >> to me in the first place.
> > > >>
> > > >> Jon
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by Nick Allen <ni...@nickallen.org>.
I agree with everything that I've read.
One of the guys from Bro had contacted me a while back, letting me know
that the packaging mechanism in Bro was ready for public consumption. I
just have not had cycles to do anything with it yet. They are not wanting
to host any of the plugins.
I thought the package mechanism requires that a package live within its own
repo (which Casey confirmed). This put me in a bind on how to tackle
this. I don't want to personally host the plugin in my own Github repo. I
would prefer that we host it in a community repo; either Bro or Metron.
Since Bro is moving away from hosting their own plugins, that leaves Metron.
It would be great if we could create a separate repo for the plugin. That
solves the challenge of using the packaging mechanism.
We do need to reconcile what is in bro/bro-plugins and what is in Metron.
There are some enhancements that I and others have made that never made it
back into Metron. They never made it back, because the original plan was
just to switch to using the plugin from bro/bro-plugins before the idea of
a packaging mechanism hit Bro. Reconciling should be fairly easy to see by
just doing a diff.
It would be great if others want to take on any of that work. I would be
glad to offer any support that you need. Thanks, Jon!
On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Ok, great.
>
> I agree, I definitely want to hear from Nick on the topic. My team is
> currently looking into enhancing the plugin as well to potentially allow
> sending to multiple clusters, investigating some issues we see when our bro
> cluster is under load, turn it into a package, etc.
>
> The work you just did was on our to do list as well so I'm very excited to
> see it come through.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com> wrote:
>
> I *think* it's possible. People do ask for mirrors of directories from
> time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If
> we
> think this is a good idea, we can pose it to INFRA as a request. I'd love
> to see us be able to use the bro packaging infrastructure and get more
> visibility for the plugin.
>
> I'd be particularly interested in Nick's opinion on this, though.
>
> On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > You can version packages -
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > versioning
> >
> > I agree that having a separate repo provided by Apache would be optimal,
> I
> > just don't know the process for that or if it was even reasonable to
> > suggest.
> >
> > Jon
> >
> > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > Looking at the bro packages, it appears that bro is expecting things to
> > be
> > > its own git repository. I wonder if we could either request INFRA
> > provide
> > > another repo for the bro-kafka plugin and integrate it into metron as a
> > git
> > > submodule *or* if we could request INFRA to create a github mirror of
> the
> > > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> > either
> > > of those options are, frankly.
> > >
> > > One thing that I didn't see is how do you specify a particular release
> of
> > > the plugin that you want to install? For us, we'd want to release the
> > > plugin along with the product. I didn't quite see how you'd push
> > releases
> > > for bro plugins.
> > >
> > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> > wrote:
> > >
> > > > So, I do agree with the concern. Is there a way to host the package
> > > > within Metron? I definitely would like to see the modifications at
> > > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for
> us
> > to
> > > > host the plugin.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > > wrote:
> > > >
> > > >> Today I was taking a look at METRON-812
> > > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > > recall
> > > >> some conversations from a while back regarding where the bro kafka
> > > plugin
> > > >> should ultimately live, and how to update it.
> > > >>
> > > >> Back in METRON-348 <https://issues.apache.org/
> jira/browse/METRON-348>
> > I
> > > >> brought up the fact that some important changes
> > > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > > >> 5348da0a5043a8353b4a0a8>
> > > >> were made to the externally hosted version of the kafka plugin, and
> > were
> > > >> never introduced to Metron's hosted version (i.e. the one we use
> > > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > > >> in vagrant when bro is installed). The conversation went down the
> > route
> > > >> of
> > > >> discussing whether or not the bro kafka plugin code should continue
> to
> > > >> live
> > > >> in Metron in the first place. Now, with METRON-812, I see us
> further
> > > >> muddying the waters of where to go for the right plugin, as our
> > version
> > > is
> > > >> still missing the public changes but adds some very important new
> > > >> functionality.
> > > >>
> > > >> I'd like to bring up the idea of using bro's packages
> > > >> <https://github.com/bro/packages> framework, released in late 2016
> > > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > > >> (additional
> > > >> documentation here <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > > >> >),
> > > >> as a potential place for this to be hosted/referenced. This is a
> > simple
> > > >> and supported method (funded by Mozilla
> > > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > > >> e-support-first-awards-made/>)
> > > >> to install and uninstall bro scripts, plugins, etc., and it also
> > allows
> > > us
> > > >> to continue to have enough control over updates to the plugin so
> that
> > it
> > > >> will not slow down Metron development by having it as a dependency
> > > >> (resolving both of Casey's concerns noted here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > > >> and I think this solution is supported by Nick's comments here
> > > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > > >> as
> > > >> well).
> > > >>
> > > >> The only thing I'm not sure about is where to host the plugin itself
> -
> > > my
> > > >> first thought would be Nick's github <https://github.com/nickwallen
> >,
> > > as
> > > >> he
> > > >> really kicked off this effort, but maybe we can think of something
> > > better.
> > > >>
> > > >> Is this approach of interest to anybody? It is extremely simple to
> > put
> > > >> together - I was able to throw one together
> > > >> <https://github.com/bro/packages/blob/master/
> jonzeolla/bro-pkg.index>
> > > and
> > > >> get it working with a fresh bro 2.5 install when attending the
> brocon
> > > talk
> > > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > > >> bro-packagemanager>
> > > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that
> introduced
> > > >> this
> > > >> to me in the first place.
> > > >>
> > > >> Jon
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Ok, great.
I agree, I definitely want to hear from Nick on the topic. My team is
currently looking into enhancing the plugin as well to potentially allow
sending to multiple clusters, investigating some issues we see when our bro
cluster is under load, turn it into a package, etc.
The work you just did was on our to do list as well so I'm very excited to
see it come through.
Jon
On Thu, Mar 30, 2017, 11:16 PM Casey Stella <ce...@gmail.com> wrote:
I *think* it's possible. People do ask for mirrors of directories from
time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If we
think this is a good idea, we can pose it to INFRA as a request. I'd love
to see us be able to use the bro packaging infrastructure and get more
visibility for the plugin.
I'd be particularly interested in Nick's opinion on this, though.
On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> You can version packages -
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> versioning
>
> I agree that having a separate repo provided by Apache would be optimal, I
> just don't know the process for that or if it was even reasonable to
> suggest.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
>
> > Looking at the bro packages, it appears that bro is expecting things to
> be
> > its own git repository. I wonder if we could either request INFRA
> provide
> > another repo for the bro-kafka plugin and integrate it into metron as a
> git
> > submodule *or* if we could request INFRA to create a github mirror of
the
> > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> either
> > of those options are, frankly.
> >
> > One thing that I didn't see is how do you specify a particular release
of
> > the plugin that you want to install? For us, we'd want to release the
> > plugin along with the product. I didn't quite see how you'd push
> releases
> > for bro plugins.
> >
> > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> wrote:
> >
> > > So, I do agree with the concern. Is there a way to host the package
> > > within Metron? I definitely would like to see the modifications at
> > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us
> to
> > > host the plugin.
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > >> Today I was taking a look at METRON-812
> > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > recall
> > >> some conversations from a while back regarding where the bro kafka
> > plugin
> > >> should ultimately live, and how to update it.
> > >>
> > >> Back in METRON-348 <https://issues.apache.org/jira/browse/METRON-348>
> I
> > >> brought up the fact that some important changes
> > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > >> 5348da0a5043a8353b4a0a8>
> > >> were made to the externally hosted version of the kafka plugin, and
> were
> > >> never introduced to Metron's hosted version (i.e. the one we use
> > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > >> in vagrant when bro is installed). The conversation went down the
> route
> > >> of
> > >> discussing whether or not the bro kafka plugin code should continue
to
> > >> live
> > >> in Metron in the first place. Now, with METRON-812, I see us further
> > >> muddying the waters of where to go for the right plugin, as our
> version
> > is
> > >> still missing the public changes but adds some very important new
> > >> functionality.
> > >>
> > >> I'd like to bring up the idea of using bro's packages
> > >> <https://github.com/bro/packages> framework, released in late 2016
> > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > >> (additional
> > >> documentation here <
> > http://bro-package-manager.readthedocs.io/en/stable/
> > >> >),
> > >> as a potential place for this to be hosted/referenced. This is a
> simple
> > >> and supported method (funded by Mozilla
> > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > >> e-support-first-awards-made/>)
> > >> to install and uninstall bro scripts, plugins, etc., and it also
> allows
> > us
> > >> to continue to have enough control over updates to the plugin so that
> it
> > >> will not slow down Metron development by having it as a dependency
> > >> (resolving both of Casey's concerns noted here
> > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > >> and I think this solution is supported by Nick's comments here
> > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > >> as
> > >> well).
> > >>
> > >> The only thing I'm not sure about is where to host the plugin itself
-
> > my
> > >> first thought would be Nick's github <https://github.com/nickwallen>,
> > as
> > >> he
> > >> really kicked off this effort, but maybe we can think of something
> > better.
> > >>
> > >> Is this approach of interest to anybody? It is extremely simple to
> put
> > >> together - I was able to throw one together
> > >> <https://github.com/bro/packages/blob/master/jonzeolla/bro-pkg.index>
> > and
> > >> get it working with a fresh bro 2.5 install when attending the brocon
> > talk
> > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > >> bro-packagemanager>
> > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that introduced
> > >> this
> > >> to me in the first place.
> > >>
> > >> Jon
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Casey Stella <ce...@gmail.com>.
I *think* it's possible. People do ask for mirrors of directories from
time to time (see https://issues.apache.org/jira/browse/INFRA-7060). If we
think this is a good idea, we can pose it to INFRA as a request. I'd love
to see us be able to use the bro packaging infrastructure and get more
visibility for the plugin.
I'd be particularly interested in Nick's opinion on this, though.
On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> You can version packages -
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> versioning
>
> I agree that having a separate repo provided by Apache would be optimal, I
> just don't know the process for that or if it was even reasonable to
> suggest.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
>
> > Looking at the bro packages, it appears that bro is expecting things to
> be
> > its own git repository. I wonder if we could either request INFRA
> provide
> > another repo for the bro-kafka plugin and integrate it into metron as a
> git
> > submodule *or* if we could request INFRA to create a github mirror of the
> > metron-sensors/bro-kafka-plugin directory. I'm not sure how viable
> either
> > of those options are, frankly.
> >
> > One thing that I didn't see is how do you specify a particular release of
> > the plugin that you want to install? For us, we'd want to release the
> > plugin along with the product. I didn't quite see how you'd push
> releases
> > for bro plugins.
> >
> > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com>
> wrote:
> >
> > > So, I do agree with the concern. Is there a way to host the package
> > > within Metron? I definitely would like to see the modifications at
> > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us
> to
> > > host the plugin.
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > >> Today I was taking a look at METRON-812
> > >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> > recall
> > >> some conversations from a while back regarding where the bro kafka
> > plugin
> > >> should ultimately live, and how to update it.
> > >>
> > >> Back in METRON-348 <https://issues.apache.org/jira/browse/METRON-348>
> I
> > >> brought up the fact that some important changes
> > >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> > >> 5348da0a5043a8353b4a0a8>
> > >> were made to the externally hosted version of the kafka plugin, and
> were
> > >> never introduced to Metron's hosted version (i.e. the one we use
> > >> <https://github.com/apache/incubator-metron/blob/master/metr
> > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > >> in vagrant when bro is installed). The conversation went down the
> route
> > >> of
> > >> discussing whether or not the bro kafka plugin code should continue to
> > >> live
> > >> in Metron in the first place. Now, with METRON-812, I see us further
> > >> muddying the waters of where to go for the right plugin, as our
> version
> > is
> > >> still missing the public changes but adds some very important new
> > >> functionality.
> > >>
> > >> I'd like to bring up the idea of using bro's packages
> > >> <https://github.com/bro/packages> framework, released in late 2016
> > >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> > >> (additional
> > >> documentation here <
> > http://bro-package-manager.readthedocs.io/en/stable/
> > >> >),
> > >> as a potential place for this to be hosted/referenced. This is a
> simple
> > >> and supported method (funded by Mozilla
> > >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> > >> e-support-first-awards-made/>)
> > >> to install and uninstall bro scripts, plugins, etc., and it also
> allows
> > us
> > >> to continue to have enough control over updates to the plugin so that
> it
> > >> will not slow down Metron development by having it as a dependency
> > >> (resolving both of Casey's concerns noted here
> > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > >> and I think this solution is supported by Nick's comments here
> > >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > >> as
> > >> well).
> > >>
> > >> The only thing I'm not sure about is where to host the plugin itself -
> > my
> > >> first thought would be Nick's github <https://github.com/nickwallen>,
> > as
> > >> he
> > >> really kicked off this effort, but maybe we can think of something
> > better.
> > >>
> > >> Is this approach of interest to anybody? It is extremely simple to
> put
> > >> together - I was able to throw one together
> > >> <https://github.com/bro/packages/blob/master/jonzeolla/bro-pkg.index>
> > and
> > >> get it working with a fresh bro 2.5 install when attending the brocon
> > talk
> > >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> > >> bro-packagemanager>
> > >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that introduced
> > >> this
> > >> to me in the first place.
> > >>
> > >> Jon
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] The bro kafka plugin
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
You can version packages -
http://bro-package-manager.readthedocs.io/en/stable/package.html#package-versioning
I agree that having a separate repo provided by Apache would be optimal, I
just don't know the process for that or if it was even reasonable to
suggest.
Jon
On Thu, Mar 30, 2017, 11:01 PM Casey Stella <ce...@gmail.com> wrote:
> Looking at the bro packages, it appears that bro is expecting things to be
> its own git repository. I wonder if we could either request INFRA provide
> another repo for the bro-kafka plugin and integrate it into metron as a git
> submodule *or* if we could request INFRA to create a github mirror of the
> metron-sensors/bro-kafka-plugin directory. I'm not sure how viable either
> of those options are, frankly.
>
> One thing that I didn't see is how do you specify a particular release of
> the plugin that you want to install? For us, we'd want to release the
> plugin along with the product. I didn't quite see how you'd push releases
> for bro plugins.
>
> On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com> wrote:
>
> > So, I do agree with the concern. Is there a way to host the package
> > within Metron? I definitely would like to see the modifications at
> > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us to
> > host the plugin.
> >
> > Thoughts?
> >
> >
> > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> >> Today I was taking a look at METRON-812
> >> <https://issues.apache.org/jira/browse/METRON-812>, which made me
> recall
> >> some conversations from a while back regarding where the bro kafka
> plugin
> >> should ultimately live, and how to update it.
> >>
> >> Back in METRON-348 <https://issues.apache.org/jira/browse/METRON-348> I
> >> brought up the fact that some important changes
> >> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
> >> 5348da0a5043a8353b4a0a8>
> >> were made to the externally hosted version of the kafka plugin, and were
> >> never introduced to Metron's hosted version (i.e. the one we use
> >> <https://github.com/apache/incubator-metron/blob/master/metr
> >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> >> in vagrant when bro is installed). The conversation went down the route
> >> of
> >> discussing whether or not the bro kafka plugin code should continue to
> >> live
> >> in Metron in the first place. Now, with METRON-812, I see us further
> >> muddying the waters of where to go for the right plugin, as our version
> is
> >> still missing the public changes but adds some very important new
> >> functionality.
> >>
> >> I'd like to bring up the idea of using bro's packages
> >> <https://github.com/bro/packages> framework, released in late 2016
> >> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> >> (additional
> >> documentation here <
> http://bro-package-manager.readthedocs.io/en/stable/
> >> >),
> >> as a potential place for this to be hosted/referenced. This is a simple
> >> and supported method (funded by Mozilla
> >> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
> >> e-support-first-awards-made/>)
> >> to install and uninstall bro scripts, plugins, etc., and it also allows
> us
> >> to continue to have enough control over updates to the plugin so that it
> >> will not slow down Metron development by having it as a dependency
> >> (resolving both of Casey's concerns noted here
> >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> >> issuetabpanels:comment-tabpanel#comment-15391865>,
> >> and I think this solution is supported by Nick's comments here
> >> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
> >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> >> issuetabpanels:comment-tabpanel#comment-15391872>
> >> as
> >> well).
> >>
> >> The only thing I'm not sure about is where to host the plugin itself -
> my
> >> first thought would be Nick's github <https://github.com/nickwallen>,
> as
> >> he
> >> really kicked off this effort, but maybe we can think of something
> better.
> >>
> >> Is this approach of interest to anybody? It is extremely simple to put
> >> together - I was able to throw one together
> >> <https://github.com/bro/packages/blob/master/jonzeolla/bro-pkg.index>
> and
> >> get it working with a fresh bro 2.5 install when attending the brocon
> talk
> >> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> >> bro-packagemanager>
> >> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that introduced
> >> this
> >> to me in the first place.
> >>
> >> Jon
> >> --
> >>
> >> Jon
> >>
> >
> >
>
--
Jon
Re: [DISCUSS] The bro kafka plugin
Posted by Casey Stella <ce...@gmail.com>.
Looking at the bro packages, it appears that bro is expecting things to be
its own git repository. I wonder if we could either request INFRA provide
another repo for the bro-kafka plugin and integrate it into metron as a git
submodule *or* if we could request INFRA to create a github mirror of the
metron-sensors/bro-kafka-plugin directory. I'm not sure how viable either
of those options are, frankly.
One thing that I didn't see is how do you specify a particular release of
the plugin that you want to install? For us, we'd want to release the
plugin along with the product. I didn't quite see how you'd push releases
for bro plugins.
On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella <ce...@gmail.com> wrote:
> So, I do agree with the concern. Is there a way to host the package
> within Metron? I definitely would like to see the modifications at
> https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us to
> host the plugin.
>
> Thoughts?
>
>
> On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
>> Today I was taking a look at METRON-812
>> <https://issues.apache.org/jira/browse/METRON-812>, which made me recall
>> some conversations from a while back regarding where the bro kafka plugin
>> should ultimately live, and how to update it.
>>
>> Back in METRON-348 <https://issues.apache.org/jira/browse/METRON-348> I
>> brought up the fact that some important changes
>> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db06
>> 5348da0a5043a8353b4a0a8>
>> were made to the externally hosted version of the kafka plugin, and were
>> never introduced to Metron's hosted version (i.e. the one we use
>> <https://github.com/apache/incubator-metron/blob/master/metr
>> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>> in vagrant when bro is installed). The conversation went down the route
>> of
>> discussing whether or not the bro kafka plugin code should continue to
>> live
>> in Metron in the first place. Now, with METRON-812, I see us further
>> muddying the waters of where to go for the right plugin, as our version is
>> still missing the public changes but adds some very important new
>> functionality.
>>
>> I'd like to bring up the idea of using bro's packages
>> <https://github.com/bro/packages> framework, released in late 2016
>> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>> (additional
>> documentation here <http://bro-package-manager.readthedocs.io/en/stable/
>> >),
>> as a potential place for this to be hosted/referenced. This is a simple
>> and supported method (funded by Mozilla
>> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-sourc
>> e-support-first-awards-made/>)
>> to install and uninstall bro scripts, plugins, etc., and it also allows us
>> to continue to have enough control over updates to the plugin so that it
>> will not slow down Metron development by having it as a dependency
>> (resolving both of Casey's concerns noted here
>> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
>> mentId=15391865&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15391865>,
>> and I think this solution is supported by Nick's comments here
>> <https://issues.apache.org/jira/browse/METRON-348?focusedCom
>> mentId=15391872&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15391872>
>> as
>> well).
>>
>> The only thing I'm not sure about is where to host the plugin itself - my
>> first thought would be Nick's github <https://github.com/nickwallen>, as
>> he
>> really kicked off this effort, but maybe we can think of something better.
>>
>> Is this approach of interest to anybody? It is extremely simple to put
>> together - I was able to throw one together
>> <https://github.com/bro/packages/blob/master/jonzeolla/bro-pkg.index> and
>> get it working with a fresh bro 2.5 install when attending the brocon talk
>> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>> bro-packagemanager>
>> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
>> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that introduced
>> this
>> to me in the first place.
>>
>> Jon
>> --
>>
>> Jon
>>
>
>
Re: [DISCUSS] The bro kafka plugin
Posted by Casey Stella <ce...@gmail.com>.
So, I do agree with the concern. Is there a way to host the package within
Metron? I definitely would like to see the modifications at
https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a
8353b4a0a8 brought back into Metron and I'd love for us to host the plugin.
Thoughts?
On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Today I was taking a look at METRON-812
> <https://issues.apache.org/jira/browse/METRON-812>, which made me recall
> some conversations from a while back regarding where the bro kafka plugin
> should ultimately live, and how to update it.
>
> Back in METRON-348 <https://issues.apache.org/jira/browse/METRON-348> I
> brought up the fact that some important changes
> <https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a
> 8353b4a0a8>
> were made to the externally hosted version of the kafka plugin, and were
> never introduced to Metron's hosted version (i.e. the one we use
> <https://github.com/apache/incubator-metron/blob/master/
> metron-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> in vagrant when bro is installed). The conversation went down the route of
> discussing whether or not the bro kafka plugin code should continue to live
> in Metron in the first place. Now, with METRON-812, I see us further
> muddying the waters of where to go for the right plugin, as our version is
> still missing the public changes but adds some very important new
> functionality.
>
> I'd like to bring up the idea of using bro's packages
> <https://github.com/bro/packages> framework, released in late 2016
> <http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> (additional
> documentation here <http://bro-package-manager.readthedocs.io/en/stable/
> >),
> as a potential place for this to be hosted/referenced. This is a simple
> and supported method (funded by Mozilla
> <https://blog.mozilla.org/blog/2015/12/10/mozilla-open-
> source-support-first-awards-made/>)
> to install and uninstall bro scripts, plugins, etc., and it also allows us
> to continue to have enough control over updates to the plugin so that it
> will not slow down Metron development by having it as a dependency
> (resolving both of Casey's concerns noted here
> <https://issues.apache.org/jira/browse/METRON-348?
> focusedCommentId=15391865&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15391865>,
> and I think this solution is supported by Nick's comments here
> <https://issues.apache.org/jira/browse/METRON-348?
> focusedCommentId=15391872&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15391872>
> as
> well).
>
> The only thing I'm not sure about is where to host the plugin itself - my
> first thought would be Nick's github <https://github.com/nickwallen>, as
> he
> really kicked off this effort, but maybe we can think of something better.
>
> Is this approach of interest to anybody? It is extremely simple to put
> together - I was able to throw one together
> <https://github.com/bro/packages/blob/master/jonzeolla/bro-pkg.index> and
> get it working with a fresh bro 2.5 install when attending the brocon talk
> <https://www.bro.org/brocon2016/brocon2016_abstracts.html#bro-
> packagemanager>
> (recording <https://www.youtube.com/watch?v=9RFfPJeGkcE>, slides
> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>) that introduced this
> to me in the first place.
>
> Jon
> --
>
> Jon
>