Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley <ma...@apache.org>]

Okay, looking at this from the perspective of making a release:

 

We have two choices:

a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of metron-bro-plugin-kafka, at the same time and using the same process (modulo the necessary) as Metron.  This is dirt simple.  

b) I or someone needs to:

    - open a jira, 

    - add the submodule to the Metron code tree, 

    - possibly (optionally) add build mechanism to the maven poms, and 

    - document as much as we think appropriate regarding what it is, how to build it, and how to update it, 

and commit that before the 0.4.2 release.

 

What is the will of the community?

Thanks,

--Matt

 

From: Nick Allen <ni...@nickallen.org>
Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
Date: Tuesday, November 28, 2017 at 9:09 AM
To: "dev@metron.apache.org" <de...@metron.apache.org>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

 

I'll add a bit to Jon's technical comments.

 

* We only created a separate repo because it was a technical requirement to leverage the bro-pkg mechanism.

* Leveraging the new bro-pkg mechanism has many advantages as outlined by Jon.

* Enabling the bro-pkg mechanism is backwards compatible.  I can install the plugin exactly how we use to.

 

While I agree with Jon's technical comments, I disagree with the non-technical ones.

 

(1) I do not want to change our release management process.  While we needed to make a new repo (a technical change), I did not want that to change how we operate as a community (our procedures, policies, versioning and release cycles).

 

(2) I see no value in adopting a separate release management process for the Bro plugin alone.  Having a separate release process does not make the plugin *more* available to the Bro community. 

 

(3) I also see no value in positioning the plugin to be spun-out of the Metron project.  It is a part of Metron and I want to see it benefit and evolve "the Apache-way".

 

In my mind, the best way to accommodate the additional repo, while minimizing changes to our release management process, is to treat the new repo as a submodule.  I fail to see significant downsides to this approach.  A few extract command-line options do not seem overly onerous to me.

 

Many thanks go to Jon for all the hard work he has put in enhancing the plugin.

 

 

On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:

In an attempt to keep this from becoming unbearably long, I will try to keep my responses short, but I would be happy to elaborate.  That's a fairly good timeline and summary, but here are some clarifications in corresponding order: 

 

- The plugin history is quite short and you can probably get a good bit of context just by looking at the commits.

- The plugin is only useful to the bro community, but it is rather popular.

- The Bro team created the idea of bro packages, which can include bro plugins, bro scripts, or BroControl plugins.  So, instead of having a 'plugins' repo, they moved to have a 'packages' repo which is by default referenced by a bro-pkg tool they wrote for package management.

- I believe I kicked this off (or at least I did in my head) when I started complaining about the plugin divergence that occurred due to the move to bro/plugins (the right move at the time), but Metron's use of a local directory that hadn't been kept up to date.  My current efforts are an attempt to make sure this doesn't happen again, and to take advantage of the bro-pkg benefits.

- The gap between ~3/31 and actual progress on 11/12 is completely on me - I had intentions of doing this work sooner but failed to do so.  

- You can most definitely still install/use the bro plugin without using bro-pkg.  In fact, the README in my PR still has the instructions on how to do so.

 

Q1:  The simple explanation is that the only thing that makes a plugin a bro package is the inclusion of a bro-pkg.meta file, and it includes a build_command which could easily be manually performed to install by hand (assuming dependencies are met).

 

I've worked with other projects that use submodules and while I'm fine discussing it, I suggest that we don't implement it.  I put together a quick example of why here[1], using the bro project as an example since it's top of mind.

 

Q2:   I think the answer to Q1 answers this.  There is absolutely nothing stopping a git clone && cd $dir && configure && make && make install, but using bro-pkg to install/load takes into account dependencies and unit tests when it is loaded (and thus fails early and more intuitively).  It only must be a separate repo (or, more technically correct, a git branch that includes only the package) because of how bro-pkg works.  If you'd like to get an idea of how this would work in application for Bro users, you can see my test instructions here (specifically step #3).  If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is just to remove user interaction, for an ansible spin-up).

 

 

1:  To clone the Bro git repo, you must run `git clone --recursive https://github.com/bro/bro` (note the --recursive).  Not too big of a deal, but requires that you remember it and existing instructions/blog posts may give users inaccurate steps.  Let's make this worse and try to checkout their latest release, v2.5.2, and automatically update the submodules appropriately via `git checkout v2.5.2 --recurse-submodules`.  This fails because aux/plugins (https://github.com/bro/plugins) was removed since their latest release.  Okay, we can work around this using `git checkout v2.5.2` and then remember to `git submodule update` every time you checkout a release or branch.  But because they have nested submodules, we actually need to run `git submodule update --recursive`.  I can't imagine opting into a workflow anything like this.  There are other options as well, such as git subtrees, but those I am less familiar with.

 

Jon

 

On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com> wrote:

I am not sure that our use of the plugin necessarily equates to it being
implicitly coupled to Metron.  It seems like the Right Thing To Do, esp.
 for an Apache project would be to make this available for use by the
greater bro community.
Unless we expect to do extensive iterative work on the plugin, which would
then make the decision to spin it out now premature.

Then again, I might be wrong ;)


On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:

[Please pardon me that the below is a little labored. I’m trying to
understand the implications for both release and use, which requires some
explanation as well as the two questions needed. Q1 and Q2 below are
probably the same question, asked in slightly different contexts. Please
consider them together.]

So this made me go back and look at the history that caused us to put the
bro plugin in a separate repo. As best I can see, this was in
https://issues.apache.org/jira/browse/METRON-813 , which cites an email
discussion thread. Also please see
https://issues.apache.org/jira/browse/METRON-883 for background on the
plugin itself.

As best I can assemble the many bits brought up in the threads, the reasons
to put it in a separate repo was:
- The plugin was thought to be useful to multiple clients of bro and kafka,
including Storm and Spark, as well as Metron.
- Originally the bro project was maintaining bro plugins and it was thought
they might adopt this one.
- Bro then formalized their plugin framework BUT dumped all plugins out of
their sphere of maintenance.
- As of 3/31/2017, Nick said that “the [bro] package mechanism requires
that a package live within its own repo”. Jon said “the bro packages model
doesn't allow colocation with anything else.”
- So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
was created a few days later. But Metron wasn’t actually modified to remove
the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
– with https://issues.apache.org/jira/browse/METRON-1309 .
- Presumably the need to have metron-bro-plugin-kafka in a separate repo
remain valid, if the bro plugin mechanism is used. But obviously there are
(non-conforming) ways to build the plugin as part of metron, and install it
in a way that works.

Q1. I think that last statement needs some explanation. Nick or Jon, can
you please expand on it, especially wrt how the end user installs the
plugin once the plugin is built the two different ways? And whether it’s
still valuable to have a separate repo for the plugin?

Nick suggests using a submodule approach to managing the bro plugin, for
Metron versioning purposes. As I understand it, this would continue the
existence of the metron-bro-plugin-kafka repo, but copy it into the metron
code tree for building, versioning, and release purposes. Git submodules
are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
We would use the submodule capability to clone the metron-bro-plugin-kafka
source code into a subdirectory of Metron at the time one clones the metron
repo. It would then be released with Metron as part of the source code
release for a given version of Metron. Part of the way submodules are
managed, is that git stores the SHA1 hash of the submodule into a file
named .gitmodules, which in turn gets saved when you do a git push. So
indeed submodules would ensure that everyone cloning a given version of
metron would get the expected “version” (sha, actually) of
metron-bro-plugin-kafka.

This sounds like a good idea, although it isn’t without cost. Submodules
impose the need for additional commands to actually get a copy of the
submodule source, and if the plugin repo advanced beyond the version in a
metron repo, it causes some ‘git status’ artifacts that could be confusing
to folks who aren’t familiar with submodules. But these can be documented.

Q2. Nick, what I’m not clear about is the process by which the
metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
developers, and (b) end users. If it “must” be in a separate repo to be
successfully built and managed by the bro plugin mechanism, does that mean
it can’t be built from the copy in the Metron source tree? Yet until
November, that’s exactly what we were doing. Do we go back to doing that?
What does that mean wrt users installing the plugin?

Thanks for your patience in reading this far.
--Matt


On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:

I agree with Nick. Since the plugin is tightly coupled with Metron why not
just pull it into the main repo and version it with the rest of the code?
Do we really need the second repo for the plug-in?

Thanks,
James



16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
>> I would suggest that we institute a release procedure for the package
>> itself, but I don't think it necessarily has to line up with metron
>> releases (happy to be persuaded otherwise). Then we can just link metron
>> to metron-bro-plugin-kafka by pointing to specific
>> metron-bro-plugin-kafka releases (git tags
>> <
http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
>> versioning>
>> ).
>> Right now, full-dev spins up against the
>> apache/metron-bro-plugin-kafka master branch, which is not a good idea
to
>> have in place for an upcoming release. That is the crux of why I think
we
>> need to finalize the move to bro 2.5.2 and the plugin packaging before
our
>> next release (working on it as we speak).
>> Jon
>
> ​I replayed Jon's comments from the other thread above.​
>
> My initial thought, is that I would not want to manage two separate
release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> + Restarting the thread to include mentors.
>>
>> The code of the 'Kafka Plugin for Bro' is now maintained in the external
>> repository that we set up a while back.
>>
>> - Metron Core: git://git.apache.org/metron.git
>> - Kafka Plugin for Bro: git://git.apache.org/
>> metron-bro-plugin-kafka.git
>>
>> (Q) Do we need to change anything in the release procedure to account
for
>> this?

-------------------
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

-- 

Jon

 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley <mf...@hortonworks.com>]

Good, I’ll build the RC tonight.  Thanks Jon.
--Matt

From: "Zeolla@GMail.com" <ze...@gmail.com>
Date: Thursday, December 7, 2017 at 12:27 PM
To: Matt Foley <mf...@hortonworks.com>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

Otto, your understanding is correct, but given Mattf's comments and my recent actions, my initial comment is no longer true - it should continue to work throughout.

Mattf, agreed.  Done<https://github.com/apache/metron-bro-plugin-kafka/tree/0.1>.

Jon

On Thu, Dec 7, 2017 at 3:14 PM Matt Foley <mf...@hortonworks.com>> wrote:
That means we can’t wait and release-vote the plugin before pushing its tag.
I recommend that you go ahead and push the 0.1 tag as soon as you push the commit.
This does not imply approval by the committee, nor does it constitute a release.
It’s just a tag.  If that tag turns out not to coincide with an approved release, that’s okay.
(But it probably will.)

Thanks,
--Matt

On 12/7/17, 12:09 PM, "Zeolla@GMail.com" <ze...@gmail.com>> wrote:

    FYI to be uber clear about the effects of what I'm doing, spinning up
    full-dev only when including the sensors will be broken on the bro plugin
    install step between when I push the changes, and when mattf pushes the 0.1
    tag to apache/metron-bro-plugin-kafka.

    Jon

    On Thu, Dec 7, 2017 at 3:05 PM Zeolla@GMail.com <ze...@gmail.com>> wrote:

    > Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
    >
    > Jon
    >
    > On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org>> wrote:
    >
    >> I can start the release process tonight.
    >>
    >>
    >>
    >> Jon, you mentioned you want to commit
    >>
    >> > https://github.com/apache/metron/pull/847 and
    >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
    >>
    >> before the release.  Is it convenient for you to do so today?
    >>
    >>
    >>
    >> Thanks,
    >>
    >> --Matt
    >>
    >>
    >>
    >> From: Nick Allen <ni...@nickallen.org>>
    >> Date: Thursday, December 7, 2017 at 10:13 AM
    >> To: "dev@metron.apache.org<ma...@metron.apache.org>" <de...@metron.apache.org>>
    >> Cc: Matt Foley <ma...@apache.org>>
    >> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
    >>
    >>
    >>
    >> I am more interested in getting a release cut.  If me moving to the (a)
    >> camp gets us to consensus and cuts a release faster, then I'll do it.
    >> Let's get this release train moving.
    >>
    >>
    >>
    >> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com>>
    >> wrote:
    >>
    >> Do we have any further discussion on this?  Pardon me if I misstate
    >> anyone's position, but it seems like we have a couple people (Otto and Jon
    >> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
    >> a
    >> section of people like myself without a particular horse in the race.
    >>
    >> It seems like we need to come to some sort of consensus so that we can get
    >> the release bus moving again, and right now it seems like (a) is gathering
    >> more explicit support.  Do we have a compelling reason to not do (a)? To
    >> be
    >> honest, my main worry is more "If we do (a) are we going to be miserable
    >> if
    >> we need to iterate or adjust?" I'm not seeing anything that suggests
    >> anything too terrible, so unless we see some more discussion, I suggest we
    >> move forward with (a).
    >>
    >>
    >> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com>>
    >> wrote:
    >>
    >> > I would prefer a, but I was initially thinking of doing the plugin first
    >> > and then get in the two PRs out to use this new tag, which are already
    >> +1'd
    >> > and just waiting on this conversation.  For reference,
    >> > https://github.com/apache/metron/pull/847 and
    >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
    >> >
    >> > Jon
    >> >
    >> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com>> wrote:
    >> >
    >> > > It seems to me, as I believe I have stated before that a) feels like
    >> the
    >> > > proper way to handle this.  It is how I have seen other projects like
    >> > NiFi
    >> > > handle things as well.
    >> > >
    >> > >
    >> > >
    >> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org<ma...@apache.org>) wrote:
    >> > >
    >> > > Okay, looking at this from the perspective of making a release:
    >> > >
    >> > >
    >> > >
    >> > > We have two choices:
    >> > >
    >> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
    >> > > metron-bro-plugin-kafka, at the same time and using the same process
    >> > > (modulo the necessary) as Metron.  This is dirt simple.
    >> > >
    >> > > b) I or someone needs to:
    >> > >
    >> > >     - open a jira,
    >> > >
    >> > >     - add the submodule to the Metron code tree,
    >> > >
    >> > >     - possibly (optionally) add build mechanism to the maven poms, and
    >> > >
    >> > >     - document as much as we think appropriate regarding what it is,
    >> how
    >> > to
    >> > > build it, and how to update it,
    >> > >
    >> > > and commit that before the 0.4.2 release.
    >> > >
    >> > >
    >> > >
    >> > > What is the will of the community?
    >> > >
    >> > > Thanks,
    >> > >
    >> > > --Matt
    >> > >
    >> > >
    >> > >
    >> > > From: Nick Allen <ni...@nickallen.org>>
    >> > > Reply-To: "dev@metron.apache.org<ma...@metron.apache.org>" <de...@metron.apache.org>>
    >> > > Date: Tuesday, November 28, 2017 at 9:09 AM
    >> > > To: "dev@metron.apache.org<ma...@metron.apache.org>" <de...@metron.apache.org>>
    >> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
    >> > Bro'
    >> > >
    >> > >
    >> > >
    >> > > I'll add a bit to Jon's technical comments.
    >> > >
    >> > >
    >> > >
    >> > > * We only created a separate repo because it was a technical
    >> requirement
    >> > to
    >> > > leverage the bro-pkg mechanism.
    >> > >
    >> > > * Leveraging the new bro-pkg mechanism has many advantages as
    >> outlined by
    >> > > Jon.
    >> > >
    >> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
    >> install
    >> > the
    >> > > plugin exactly how we use to.
    >> > >
    >> > >
    >> > >
    >> > > While I agree with Jon's technical comments, I disagree with the
    >> > > non-technical ones.
    >> > >
    >> > >
    >> > >
    >> > > (1) I do not want to change our release management process. While we
    >> > needed
    >> > > to make a new repo (a technical change), I did not want that to change
    >> > how
    >> > > we operate as a community (our procedures, policies, versioning and
    >> > release
    >> > > cycles).
    >> > >
    >> > >
    >> > >
    >> > > (2) I see no value in adopting a separate release management process
    >> for
    >> > > the Bro plugin alone. Having a separate release process does not make
    >> the
    >> > > plugin *more* available to the Bro community.
    >> > >
    >> > >
    >> > >
    >> > > (3) I also see no value in positioning the plugin to be spun-out of
    >> the
    >> > > Metron project. It is a part of Metron and I want to see it benefit
    >> and
    >> > > evolve "the Apache-way".
    >> > >
    >> > >
    >> > >
    >> > > In my mind, the best way to accommodate the additional repo, while
    >> > > minimizing changes to our release management process, is to treat the
    >> new
    >> > > repo as a submodule. I fail to see significant downsides to this
    >> > approach.
    >> > > A few extract command-line options do not seem overly onerous to me.
    >> > >
    >> > >
    >> > >
    >> > > Many thanks go to Jon for all the hard work he has put in enhancing
    >> the
    >> > > plugin.
    >> > >
    >> > >
    >> > >
    >> > >
    >> > >
    >> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>>
    >> > > wrote:
    >> > >
    >> > > In an attempt to keep this from becoming unbearably long, I will try
    >> to
    >> > > keep my responses short, but I would be happy to elaborate. That's a
    >> > fairly
    >> > > good timeline and summary, but here are some clarifications in
    >> > > corresponding order:
    >> > >
    >> > >
    >> > >
    >> > > - The plugin history is quite short and you can probably get a good
    >> bit
    >> > of
    >> > > context just by looking at the commits.
    >> > >
    >> > > - The plugin is only useful to the bro community, but it is rather
    >> > popular.
    >> > >
    >> > > - The Bro team created the idea of bro packages, which can include bro
    >> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
    >> > > 'plugins' repo, they moved to have a 'packages' repo which is by
    >> default
    >> > > referenced by a bro-pkg tool they wrote for package management.
    >> > >
    >> > > - I believe I kicked this off (or at least I did in my head) when I
    >> > started
    >> > > complaining about the plugin divergence that occurred due to the move
    >> to
    >> > > bro/plugins (the right move at the time), but Metron's use of a local
    >> > > directory that hadn't been kept up to date. My current efforts are an
    >> > > attempt to make sure this doesn't happen again, and to take advantage
    >> of
    >> > > the bro-pkg benefits.
    >> > >
    >> > > - The gap between ~3/31 and actual progress on 11/12 is completely on
    >> me
    >> > -
    >> > > I had intentions of doing this work sooner but failed to do so.
    >> > >
    >> > > - You can most definitely still install/use the bro plugin without
    >> using
    >> > > bro-pkg. In fact, the README in my PR still has the instructions on
    >> how
    >> > to
    >> > > do so.
    >> > >
    >> > >
    >> > >
    >> > > Q1: The simple explanation is that the only thing that makes a plugin
    >> a
    >> > bro
    >> > > package is the inclusion of a bro-pkg.meta file, and it includes a
    >> > > build_command which could easily be manually performed to install by
    >> hand
    >> > > (assuming dependencies are met).
    >> > >
    >> > >
    >> > >
    >> > > I've worked with other projects that use submodules and while I'm fine
    >> > > discussing it, I suggest that we don't implement it. I put together a
    >> > quick
    >> > > example of why here[1], using the bro project as an example since it's
    >> > top
    >> > > of mind.
    >> > >
    >> > >
    >> > >
    >> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
    >> > > stopping a git clone && cd $dir && configure && make && make install,
    >> but
    >> > > using bro-pkg to install/load takes into account dependencies and unit
    >> > > tests when it is loaded (and thus fails early and more intuitively).
    >> It
    >> > > only must be a separate repo (or, more technically correct, a git
    >> branch
    >> > > that includes only the package) because of how bro-pkg works. If you'd
    >> > like
    >> > > to get an idea of how this would work in application for Bro users,
    >> you
    >> > can
    >> > > see my test instructions here (specifically step #3). If a 0.1 tag
    >> gets
    >> > > pushed to apache/metron-bro-plugin-kafka, the command could be
    >> `bro-pkg
    >> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
    >> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the
    >> --force is
    >> > > just to remove user interaction, for an ansible spin-up).
    >> > >
    >> > >
    >> > >
    >> > >
    >> > >
    >> > > 1: To clone the Bro git repo, you must run `git clone --recursive
    >> > > https://github.com/bro/bro`<https://github.com/bro/bro> <https://github.com/bro/bro> <
    >> https://github.com/bro/bro> (note the
    >> > > --recursive). Not too big of a deal,
    >> > > but requires that you remember it and existing instructions/blog posts
    >> > may
    >> > > give users inaccurate steps. Let's make this worse and try to checkout
    >> > > their latest release, v2.5.2, and automatically update the submodules
    >> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
    >> fails
    >> > > because aux/plugins (https://github.com/bro/plugins) was removed
    >> since
    >> > > their latest release. Okay, we can work around this using `git
    >> checkout
    >> > > v2.5.2` and then remember to `git submodule update` every time you
    >> > checkout
    >> > > a release or branch. But because they have nested submodules, we
    >> actually
    >> > > need to run `git submodule update --recursive`. I can't imagine opting
    >> > into
    >> > > a workflow anything like this. There are other options as well, such
    >> as
    >> > git
    >> > > subtrees, but those I am less familiar with.
    >> > >
    >> > >
    >> > >
    >> > > Jon
    >> > >
    >> > >
    >> > >
    >> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>>
    >> > > wrote:
    >> > >
    >> > > I am not sure that our use of the plugin necessarily equates to it
    >> being
    >> > > implicitly coupled to Metron. It seems like the Right Thing To Do,
    >> esp.
    >> > > for an Apache project would be to make this available for use by the
    >> > > greater bro community.
    >> > > Unless we expect to do extensive iterative work on the plugin, which
    >> > would
    >> > > then make the decision to spin it out now premature.
    >> > >
    >> > > Then again, I might be wrong ;)
    >> > >
    >> > >
    >> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org<ma...@apache.org>)
    >> wrote:
    >> > >
    >> > > [Please pardon me that the below is a little labored. I’m trying to
    >> > > understand the implications for both release and use, which requires
    >> some
    >> > > explanation as well as the two questions needed. Q1 and Q2 below are
    >> > > probably the same question, asked in slightly different contexts.
    >> Please
    >> > > consider them together.]
    >> > >
    >> > > So this made me go back and look at the history that caused us to put
    >> the
    >> > > bro plugin in a separate repo. As best I can see, this was in
    >> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
    >> email
    >> > > discussion thread. Also please see
    >> > > https://issues.apache.org/jira/browse/METRON-883 for background on
    >> the
    >> > > plugin itself.
    >> > >
    >> > > As best I can assemble the many bits brought up in the threads, the
    >> > reasons
    >> > > to put it in a separate repo was:
    >> > > - The plugin was thought to be useful to multiple clients of bro and
    >> > kafka,
    >> > > including Storm and Spark, as well as Metron.
    >> > > - Originally the bro project was maintaining bro plugins and it was
    >> > thought
    >> > > they might adopt this one.
    >> > > - Bro then formalized their plugin framework BUT dumped all plugins
    >> out
    >> > of
    >> > > their sphere of maintenance.
    >> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism
    >> requires
    >> > > that a package live within its own repo”. Jon said “the bro packages
    >> > model
    >> > > doesn't allow colocation with anything else.”
    >> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
    >> repo
    >> > > was created a few days later. But Metron wasn’t actually modified to
    >> > remove
    >> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
    >> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
    >> > ago!
    >> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
    >> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
    >> repo
    >> > > remain valid, if the bro plugin mechanism is used. But obviously there
    >> > are
    >> > > (non-conforming) ways to build the plugin as part of metron, and
    >> install
    >> > it
    >> > > in a way that works.
    >> > >
    >> > > Q1. I think that last statement needs some explanation. Nick or Jon,
    >> can
    >> > > you please expand on it, especially wrt how the end user installs the
    >> > > plugin once the plugin is built the two different ways? And whether
    >> it’s
    >> > > still valuable to have a separate repo for the plugin?
    >> > >
    >> > > Nick suggests using a submodule approach to managing the bro plugin,
    >> for
    >> > > Metron versioning purposes. As I understand it, this would continue
    >> the
    >> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
    >> > metron
    >> > > code tree for building, versioning, and release purposes. Git
    >> submodules
    >> > > are documented here:
    >> https://git-scm.com/book/en/v2/Git-Tools-Submodules
    >> > .
    >> > > We would use the submodule capability to clone the
    >> > metron-bro-plugin-kafka
    >> > > source code into a subdirectory of Metron at the time one clones the
    >> > metron
    >> > > repo. It would then be released with Metron as part of the source code
    >> > > release for a given version of Metron. Part of the way submodules are
    >> > > managed, is that git stores the SHA1 hash of the submodule into a file
    >> > > named .gitmodules, which in turn gets saved when you do a git push. So
    >> > > indeed submodules would ensure that everyone cloning a given version
    >> of
    >> > > metron would get the expected “version” (sha, actually) of
    >> > > metron-bro-plugin-kafka.
    >> > >
    >> > > This sounds like a good idea, although it isn’t without cost.
    >> Submodules
    >> > > impose the need for additional commands to actually get a copy of the
    >> > > submodule source, and if the plugin repo advanced beyond the version
    >> in a
    >> > > metron repo, it causes some ‘git status’ artifacts that could be
    >> > confusing
    >> > > to folks who aren’t familiar with submodules. But these can be
    >> > documented.
    >> > >
    >> > > Q2. Nick, what I’m not clear about is the process by which the
    >> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
    >> > > developers, and (b) end users. If it “must” be in a separate repo to
    >> be
    >> > > successfully built and managed by the bro plugin mechanism, does that
    >> > mean
    >> > > it can’t be built from the copy in the Metron source tree? Yet until
    >> > > November, that’s exactly what we were doing. Do we go back to doing
    >> that?
    >> > > What does that mean wrt users installing the plugin?
    >> > >
    >> > > Thanks for your patience in reading this far.
    >> > > --Matt
    >> > >
    >> > >
    >> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org>> wrote:
    >> > >
    >> > > I agree with Nick. Since the plugin is tightly coupled with Metron why
    >> > not
    >> > > just pull it into the main repo and version it with the rest of the
    >> code?
    >> > > Do we really need the second repo for the plug-in?
    >> > >
    >> > > Thanks,
    >> > > James
    >> > >
    >> > >
    >> > >
    >> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>>:
    >> > > >> I would suggest that we institute a release procedure for the
    >> package
    >> > > >> itself, but I don't think it necessarily has to line up with metron
    >> > > >> releases (happy to be persuaded otherwise). Then we can just link
    >> > metron
    >> > > >> to metron-bro-plugin-kafka by pointing to specific
    >> > > >> metron-bro-plugin-kafka releases (git tags
    >> > > >> <
    >> > > http://bro-package-manager.readthedocs.io/en/stable/
    >> > package.html#package-
    >> > > >> versioning>
    >> > > >> ).
    >> > > >> Right now, full-dev spins up against the
    >> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
    >> idea
    >> > > to
    >> > > >> have in place for an upcoming release. That is the crux of why I
    >> think
    >> > > we
    >> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
    >> before
    >> > > our
    >> > > >> next release (working on it as we speak).
    >> > > >> Jon
    >> > > >
    >> > > > ​I replayed Jon's comments from the other thread above.​
    >> > > >
    >> > > > My initial thought, is that I would not want to manage two separate
    >> > > release
    >> > > > processes. I don't want to have a roll call, cut release candidates
    >> and
    >> > > > test both.
    >> > > >
    >> > > > I was thinking we would just need to change some of the
    >> > behind-the-scenes
    >> > > > processes handled by the release manager. This is one area where I
    >> had
    >> > > > thought using a submodule in Git would help.
    >> > > >
    >> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>>
    >> > wrote:
    >> > > >
    >> > > >> + Restarting the thread to include mentors.
    >> > > >>
    >> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
    >> > external
    >> > > >> repository that we set up a while back.
    >> > > >>
    >> > > >> - Metron Core: git://git.apache.org/metron.git<http://git.apache.org/metron.git>
    >> > > >> - Kafka Plugin for Bro: git://git.apache.org/<http://git.apache.org/>
    >> > > >> metron-bro-plugin-kafka.git
    >> > > >>
    >> > > >> (Q) Do we need to change anything in the release procedure to
    >> account
    >> > > for
    >> > > >> this?
    >> > >
    >> > > -------------------
    >> > > Thank you,
    >> > >
    >> > > James Sirota
    >> > > PMC- Apache Metron
    >> > > jsirota AT apache DOT org
    >> > >
    >> > > --
    >> > >
    >> > > Jon
    >> > >
    >> > --
    >> >
    >> > Jon
    >> >
    >>
    >>
    >>
    >> --
    >
    > Jon
    >
    --

    Jon

--

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley <mf...@hortonworks.com>]

That means we can’t wait and release-vote the plugin before pushing its tag.
I recommend that you go ahead and push the 0.1 tag as soon as you push the commit.
This does not imply approval by the committee, nor does it constitute a release.  
It’s just a tag.  If that tag turns out not to coincide with an approved release, that’s okay. 
(But it probably will.)

Thanks,
--Matt 

On 12/7/17, 12:09 PM, "Zeolla@GMail.com" <ze...@gmail.com> wrote:

    FYI to be uber clear about the effects of what I'm doing, spinning up
    full-dev only when including the sensors will be broken on the bro plugin
    install step between when I push the changes, and when mattf pushes the 0.1
    tag to apache/metron-bro-plugin-kafka.
    
    Jon
    
    On Thu, Dec 7, 2017 at 3:05 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
    
    > Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
    >
    > Jon
    >
    > On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote:
    >
    >> I can start the release process tonight.
    >>
    >>
    >>
    >> Jon, you mentioned you want to commit
    >>
    >> > https://github.com/apache/metron/pull/847 and
    >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
    >>
    >> before the release.  Is it convenient for you to do so today?
    >>
    >>
    >>
    >> Thanks,
    >>
    >> --Matt
    >>
    >>
    >>
    >> From: Nick Allen <ni...@nickallen.org>
    >> Date: Thursday, December 7, 2017 at 10:13 AM
    >> To: "dev@metron.apache.org" <de...@metron.apache.org>
    >> Cc: Matt Foley <ma...@apache.org>
    >> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
    >>
    >>
    >>
    >> I am more interested in getting a release cut.  If me moving to the (a)
    >> camp gets us to consensus and cuts a release faster, then I'll do it.
    >> Let's get this release train moving.
    >>
    >>
    >>
    >> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com>
    >> wrote:
    >>
    >> Do we have any further discussion on this?  Pardon me if I misstate
    >> anyone's position, but it seems like we have a couple people (Otto and Jon
    >> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
    >> a
    >> section of people like myself without a particular horse in the race.
    >>
    >> It seems like we need to come to some sort of consensus so that we can get
    >> the release bus moving again, and right now it seems like (a) is gathering
    >> more explicit support.  Do we have a compelling reason to not do (a)? To
    >> be
    >> honest, my main worry is more "If we do (a) are we going to be miserable
    >> if
    >> we need to iterate or adjust?" I'm not seeing anything that suggests
    >> anything too terrible, so unless we see some more discussion, I suggest we
    >> move forward with (a).
    >>
    >>
    >> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com>
    >> wrote:
    >>
    >> > I would prefer a, but I was initially thinking of doing the plugin first
    >> > and then get in the two PRs out to use this new tag, which are already
    >> +1'd
    >> > and just waiting on this conversation.  For reference,
    >> > https://github.com/apache/metron/pull/847 and
    >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
    >> >
    >> > Jon
    >> >
    >> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
    >> >
    >> > > It seems to me, as I believe I have stated before that a) feels like
    >> the
    >> > > proper way to handle this.  It is how I have seen other projects like
    >> > NiFi
    >> > > handle things as well.
    >> > >
    >> > >
    >> > >
    >> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
    >> > >
    >> > > Okay, looking at this from the perspective of making a release:
    >> > >
    >> > >
    >> > >
    >> > > We have two choices:
    >> > >
    >> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
    >> > > metron-bro-plugin-kafka, at the same time and using the same process
    >> > > (modulo the necessary) as Metron.  This is dirt simple.
    >> > >
    >> > > b) I or someone needs to:
    >> > >
    >> > >     - open a jira,
    >> > >
    >> > >     - add the submodule to the Metron code tree,
    >> > >
    >> > >     - possibly (optionally) add build mechanism to the maven poms, and
    >> > >
    >> > >     - document as much as we think appropriate regarding what it is,
    >> how
    >> > to
    >> > > build it, and how to update it,
    >> > >
    >> > > and commit that before the 0.4.2 release.
    >> > >
    >> > >
    >> > >
    >> > > What is the will of the community?
    >> > >
    >> > > Thanks,
    >> > >
    >> > > --Matt
    >> > >
    >> > >
    >> > >
    >> > > From: Nick Allen <ni...@nickallen.org>
    >> > > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
    >> > > Date: Tuesday, November 28, 2017 at 9:09 AM
    >> > > To: "dev@metron.apache.org" <de...@metron.apache.org>
    >> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
    >> > Bro'
    >> > >
    >> > >
    >> > >
    >> > > I'll add a bit to Jon's technical comments.
    >> > >
    >> > >
    >> > >
    >> > > * We only created a separate repo because it was a technical
    >> requirement
    >> > to
    >> > > leverage the bro-pkg mechanism.
    >> > >
    >> > > * Leveraging the new bro-pkg mechanism has many advantages as
    >> outlined by
    >> > > Jon.
    >> > >
    >> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
    >> install
    >> > the
    >> > > plugin exactly how we use to.
    >> > >
    >> > >
    >> > >
    >> > > While I agree with Jon's technical comments, I disagree with the
    >> > > non-technical ones.
    >> > >
    >> > >
    >> > >
    >> > > (1) I do not want to change our release management process. While we
    >> > needed
    >> > > to make a new repo (a technical change), I did not want that to change
    >> > how
    >> > > we operate as a community (our procedures, policies, versioning and
    >> > release
    >> > > cycles).
    >> > >
    >> > >
    >> > >
    >> > > (2) I see no value in adopting a separate release management process
    >> for
    >> > > the Bro plugin alone. Having a separate release process does not make
    >> the
    >> > > plugin *more* available to the Bro community.
    >> > >
    >> > >
    >> > >
    >> > > (3) I also see no value in positioning the plugin to be spun-out of
    >> the
    >> > > Metron project. It is a part of Metron and I want to see it benefit
    >> and
    >> > > evolve "the Apache-way".
    >> > >
    >> > >
    >> > >
    >> > > In my mind, the best way to accommodate the additional repo, while
    >> > > minimizing changes to our release management process, is to treat the
    >> new
    >> > > repo as a submodule. I fail to see significant downsides to this
    >> > approach.
    >> > > A few extract command-line options do not seem overly onerous to me.
    >> > >
    >> > >
    >> > >
    >> > > Many thanks go to Jon for all the hard work he has put in enhancing
    >> the
    >> > > plugin.
    >> > >
    >> > >
    >> > >
    >> > >
    >> > >
    >> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
    >> > > wrote:
    >> > >
    >> > > In an attempt to keep this from becoming unbearably long, I will try
    >> to
    >> > > keep my responses short, but I would be happy to elaborate. That's a
    >> > fairly
    >> > > good timeline and summary, but here are some clarifications in
    >> > > corresponding order:
    >> > >
    >> > >
    >> > >
    >> > > - The plugin history is quite short and you can probably get a good
    >> bit
    >> > of
    >> > > context just by looking at the commits.
    >> > >
    >> > > - The plugin is only useful to the bro community, but it is rather
    >> > popular.
    >> > >
    >> > > - The Bro team created the idea of bro packages, which can include bro
    >> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
    >> > > 'plugins' repo, they moved to have a 'packages' repo which is by
    >> default
    >> > > referenced by a bro-pkg tool they wrote for package management.
    >> > >
    >> > > - I believe I kicked this off (or at least I did in my head) when I
    >> > started
    >> > > complaining about the plugin divergence that occurred due to the move
    >> to
    >> > > bro/plugins (the right move at the time), but Metron's use of a local
    >> > > directory that hadn't been kept up to date. My current efforts are an
    >> > > attempt to make sure this doesn't happen again, and to take advantage
    >> of
    >> > > the bro-pkg benefits.
    >> > >
    >> > > - The gap between ~3/31 and actual progress on 11/12 is completely on
    >> me
    >> > -
    >> > > I had intentions of doing this work sooner but failed to do so.
    >> > >
    >> > > - You can most definitely still install/use the bro plugin without
    >> using
    >> > > bro-pkg. In fact, the README in my PR still has the instructions on
    >> how
    >> > to
    >> > > do so.
    >> > >
    >> > >
    >> > >
    >> > > Q1: The simple explanation is that the only thing that makes a plugin
    >> a
    >> > bro
    >> > > package is the inclusion of a bro-pkg.meta file, and it includes a
    >> > > build_command which could easily be manually performed to install by
    >> hand
    >> > > (assuming dependencies are met).
    >> > >
    >> > >
    >> > >
    >> > > I've worked with other projects that use submodules and while I'm fine
    >> > > discussing it, I suggest that we don't implement it. I put together a
    >> > quick
    >> > > example of why here[1], using the bro project as an example since it's
    >> > top
    >> > > of mind.
    >> > >
    >> > >
    >> > >
    >> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
    >> > > stopping a git clone && cd $dir && configure && make && make install,
    >> but
    >> > > using bro-pkg to install/load takes into account dependencies and unit
    >> > > tests when it is loaded (and thus fails early and more intuitively).
    >> It
    >> > > only must be a separate repo (or, more technically correct, a git
    >> branch
    >> > > that includes only the package) because of how bro-pkg works. If you'd
    >> > like
    >> > > to get an idea of how this would work in application for Bro users,
    >> you
    >> > can
    >> > > see my test instructions here (specifically step #3). If a 0.1 tag
    >> gets
    >> > > pushed to apache/metron-bro-plugin-kafka, the command could be
    >> `bro-pkg
    >> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
    >> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the
    >> --force is
    >> > > just to remove user interaction, for an ansible spin-up).
    >> > >
    >> > >
    >> > >
    >> > >
    >> > >
    >> > > 1: To clone the Bro git repo, you must run `git clone --recursive
    >> > > https://github.com/bro/bro` <https://github.com/bro/bro> <
    >> https://github.com/bro/bro> (note the
    >> > > --recursive). Not too big of a deal,
    >> > > but requires that you remember it and existing instructions/blog posts
    >> > may
    >> > > give users inaccurate steps. Let's make this worse and try to checkout
    >> > > their latest release, v2.5.2, and automatically update the submodules
    >> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
    >> fails
    >> > > because aux/plugins (https://github.com/bro/plugins) was removed
    >> since
    >> > > their latest release. Okay, we can work around this using `git
    >> checkout
    >> > > v2.5.2` and then remember to `git submodule update` every time you
    >> > checkout
    >> > > a release or branch. But because they have nested submodules, we
    >> actually
    >> > > need to run `git submodule update --recursive`. I can't imagine opting
    >> > into
    >> > > a workflow anything like this. There are other options as well, such
    >> as
    >> > git
    >> > > subtrees, but those I am less familiar with.
    >> > >
    >> > >
    >> > >
    >> > > Jon
    >> > >
    >> > >
    >> > >
    >> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
    >> > > wrote:
    >> > >
    >> > > I am not sure that our use of the plugin necessarily equates to it
    >> being
    >> > > implicitly coupled to Metron. It seems like the Right Thing To Do,
    >> esp.
    >> > > for an Apache project would be to make this available for use by the
    >> > > greater bro community.
    >> > > Unless we expect to do extensive iterative work on the plugin, which
    >> > would
    >> > > then make the decision to spin it out now premature.
    >> > >
    >> > > Then again, I might be wrong ;)
    >> > >
    >> > >
    >> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org)
    >> wrote:
    >> > >
    >> > > [Please pardon me that the below is a little labored. I’m trying to
    >> > > understand the implications for both release and use, which requires
    >> some
    >> > > explanation as well as the two questions needed. Q1 and Q2 below are
    >> > > probably the same question, asked in slightly different contexts.
    >> Please
    >> > > consider them together.]
    >> > >
    >> > > So this made me go back and look at the history that caused us to put
    >> the
    >> > > bro plugin in a separate repo. As best I can see, this was in
    >> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
    >> email
    >> > > discussion thread. Also please see
    >> > > https://issues.apache.org/jira/browse/METRON-883 for background on
    >> the
    >> > > plugin itself.
    >> > >
    >> > > As best I can assemble the many bits brought up in the threads, the
    >> > reasons
    >> > > to put it in a separate repo was:
    >> > > - The plugin was thought to be useful to multiple clients of bro and
    >> > kafka,
    >> > > including Storm and Spark, as well as Metron.
    >> > > - Originally the bro project was maintaining bro plugins and it was
    >> > thought
    >> > > they might adopt this one.
    >> > > - Bro then formalized their plugin framework BUT dumped all plugins
    >> out
    >> > of
    >> > > their sphere of maintenance.
    >> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism
    >> requires
    >> > > that a package live within its own repo”. Jon said “the bro packages
    >> > model
    >> > > doesn't allow colocation with anything else.”
    >> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
    >> repo
    >> > > was created a few days later. But Metron wasn’t actually modified to
    >> > remove
    >> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
    >> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
    >> > ago!
    >> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
    >> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
    >> repo
    >> > > remain valid, if the bro plugin mechanism is used. But obviously there
    >> > are
    >> > > (non-conforming) ways to build the plugin as part of metron, and
    >> install
    >> > it
    >> > > in a way that works.
    >> > >
    >> > > Q1. I think that last statement needs some explanation. Nick or Jon,
    >> can
    >> > > you please expand on it, especially wrt how the end user installs the
    >> > > plugin once the plugin is built the two different ways? And whether
    >> it’s
    >> > > still valuable to have a separate repo for the plugin?
    >> > >
    >> > > Nick suggests using a submodule approach to managing the bro plugin,
    >> for
    >> > > Metron versioning purposes. As I understand it, this would continue
    >> the
    >> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
    >> > metron
    >> > > code tree for building, versioning, and release purposes. Git
    >> submodules
    >> > > are documented here:
    >> https://git-scm.com/book/en/v2/Git-Tools-Submodules
    >> > .
    >> > > We would use the submodule capability to clone the
    >> > metron-bro-plugin-kafka
    >> > > source code into a subdirectory of Metron at the time one clones the
    >> > metron
    >> > > repo. It would then be released with Metron as part of the source code
    >> > > release for a given version of Metron. Part of the way submodules are
    >> > > managed, is that git stores the SHA1 hash of the submodule into a file
    >> > > named .gitmodules, which in turn gets saved when you do a git push. So
    >> > > indeed submodules would ensure that everyone cloning a given version
    >> of
    >> > > metron would get the expected “version” (sha, actually) of
    >> > > metron-bro-plugin-kafka.
    >> > >
    >> > > This sounds like a good idea, although it isn’t without cost.
    >> Submodules
    >> > > impose the need for additional commands to actually get a copy of the
    >> > > submodule source, and if the plugin repo advanced beyond the version
    >> in a
    >> > > metron repo, it causes some ‘git status’ artifacts that could be
    >> > confusing
    >> > > to folks who aren’t familiar with submodules. But these can be
    >> > documented.
    >> > >
    >> > > Q2. Nick, what I’m not clear about is the process by which the
    >> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
    >> > > developers, and (b) end users. If it “must” be in a separate repo to
    >> be
    >> > > successfully built and managed by the bro plugin mechanism, does that
    >> > mean
    >> > > it can’t be built from the copy in the Metron source tree? Yet until
    >> > > November, that’s exactly what we were doing. Do we go back to doing
    >> that?
    >> > > What does that mean wrt users installing the plugin?
    >> > >
    >> > > Thanks for your patience in reading this far.
    >> > > --Matt
    >> > >
    >> > >
    >> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
    >> > >
    >> > > I agree with Nick. Since the plugin is tightly coupled with Metron why
    >> > not
    >> > > just pull it into the main repo and version it with the rest of the
    >> code?
    >> > > Do we really need the second repo for the plug-in?
    >> > >
    >> > > Thanks,
    >> > > James
    >> > >
    >> > >
    >> > >
    >> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
    >> > > >> I would suggest that we institute a release procedure for the
    >> package
    >> > > >> itself, but I don't think it necessarily has to line up with metron
    >> > > >> releases (happy to be persuaded otherwise). Then we can just link
    >> > metron
    >> > > >> to metron-bro-plugin-kafka by pointing to specific
    >> > > >> metron-bro-plugin-kafka releases (git tags
    >> > > >> <
    >> > > http://bro-package-manager.readthedocs.io/en/stable/
    >> > package.html#package-
    >> > > >> versioning>
    >> > > >> ).
    >> > > >> Right now, full-dev spins up against the
    >> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
    >> idea
    >> > > to
    >> > > >> have in place for an upcoming release. That is the crux of why I
    >> think
    >> > > we
    >> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
    >> before
    >> > > our
    >> > > >> next release (working on it as we speak).
    >> > > >> Jon
    >> > > >
    >> > > > ​I replayed Jon's comments from the other thread above.​
    >> > > >
    >> > > > My initial thought, is that I would not want to manage two separate
    >> > > release
    >> > > > processes. I don't want to have a roll call, cut release candidates
    >> and
    >> > > > test both.
    >> > > >
    >> > > > I was thinking we would just need to change some of the
    >> > behind-the-scenes
    >> > > > processes handled by the release manager. This is one area where I
    >> had
    >> > > > thought using a submodule in Git would help.
    >> > > >
    >> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
    >> > wrote:
    >> > > >
    >> > > >> + Restarting the thread to include mentors.
    >> > > >>
    >> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
    >> > external
    >> > > >> repository that we set up a while back.
    >> > > >>
    >> > > >> - Metron Core: git://git.apache.org/metron.git
    >> > > >> - Kafka Plugin for Bro: git://git.apache.org/
    >> > > >> metron-bro-plugin-kafka.git
    >> > > >>
    >> > > >> (Q) Do we need to change anything in the release procedure to
    >> account
    >> > > for
    >> > > >> this?
    >> > >
    >> > > -------------------
    >> > > Thank you,
    >> > >
    >> > > James Sirota
    >> > > PMC- Apache Metron
    >> > > jsirota AT apache DOT org
    >> > >
    >> > > --
    >> > >
    >> > > Jon
    >> > >
    >> > --
    >> >
    >> > Jon
    >> >
    >>
    >>
    >>
    >> --
    >
    > Jon
    >
    -- 
    
    Jon
    

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Otto Fowler <ot...@gmail.com>]

You and Matt should coordinate sending a mail to the dev list with a heads
up, starting and done.

I think you mean that:

Between XXXXX  and YYYYYYY, if you do a fetch apache && checkout -b foo
apache/master and then
do vagrant up with the sensors enabled it will fail.  Right?



On December 7, 2017 at 15:09:52, Zeolla@GMail.com (zeolla@gmail.com) wrote:

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM Zeolla@GMail.com <ze...@gmail.com> wrote:

> Sounds good. Yes Matt, I will handle my parts now. Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release. Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen <ni...@nickallen.org>
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" <de...@metron.apache.org>
>> Cc: Matt Foley <ma...@apache.org>
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
Bro'
>>
>>
>>
>> I am more interested in getting a release cut. If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com>
>> wrote:
>>
>> Do we have any further discussion on this? Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and
Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and
presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can
get
>> the release bus moving again, and right now it seems like (a) is
gathering
>> more explicit support. Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest
we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin
first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation. For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com>
wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this. It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org)
wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron. This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > > - open a jira,
>> > >
>> > > - add the submodule to the Metron code tree,
>> > >
>> > > - possibly (optionally) add build mechanism to the maven poms, and
>> > >
>> > > - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen <ni...@nickallen.org>
>> > > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
>> > > Date: Tuesday, November 28, 2017 at 9:09 AM
>> > > To: "dev@metron.apache.org" <de...@metron.apache.org>
>> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin
for
>> > Bro'
>> > >
>> > >
>> > >
>> > > I'll add a bit to Jon's technical comments.
>> > >
>> > >
>> > >
>> > > * We only created a separate repo because it was a technical
>> requirement
>> > to
>> > > leverage the bro-pkg mechanism.
>> > >
>> > > * Leveraging the new bro-pkg mechanism has many advantages as
>> outlined by
>> > > Jon.
>> > >
>> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
>> install
>> > the
>> > > plugin exactly how we use to.
>> > >
>> > >
>> > >
>> > > While I agree with Jon's technical comments, I disagree with the
>> > > non-technical ones.
>> > >
>> > >
>> > >
>> > > (1) I do not want to change our release management process. While we
>> > needed
>> > > to make a new repo (a technical change), I did not want that to
change
>> > how
>> > > we operate as a community (our procedures, policies, versioning and
>> > release
>> > > cycles).
>> > >
>> > >
>> > >
>> > > (2) I see no value in adopting a separate release management process
>> for
>> > > the Bro plugin alone. Having a separate release process does not
make
>> the
>> > > plugin *more* available to the Bro community.
>> > >
>> > >
>> > >
>> > > (3) I also see no value in positioning the plugin to be spun-out of
>> the
>> > > Metron project. It is a part of Metron and I want to see it benefit
>> and
>> > > evolve "the Apache-way".
>> > >
>> > >
>> > >
>> > > In my mind, the best way to accommodate the additional repo, while
>> > > minimizing changes to our release management process, is to treat
the
>> new
>> > > repo as a submodule. I fail to see significant downsides to this
>> > approach.
>> > > A few extract command-line options do not seem overly onerous to me.
>> > >
>> > >
>> > >
>> > > Many thanks go to Jon for all the hard work he has put in enhancing
>> the
>> > > plugin.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>

>> > > wrote:
>> > >
>> > > In an attempt to keep this from becoming unbearably long, I will try
>> to
>> > > keep my responses short, but I would be happy to elaborate. That's a
>> > fairly
>> > > good timeline and summary, but here are some clarifications in
>> > > corresponding order:
>> > >
>> > >
>> > >
>> > > - The plugin history is quite short and you can probably get a good
>> bit
>> > of
>> > > context just by looking at the commits.
>> > >
>> > > - The plugin is only useful to the bro community, but it is rather
>> > popular.
>> > >
>> > > - The Bro team created the idea of bro packages, which can include
bro
>> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
>> > > 'plugins' repo, they moved to have a 'packages' repo which is by
>> default
>> > > referenced by a bro-pkg tool they wrote for package management.
>> > >
>> > > - I believe I kicked this off (or at least I did in my head) when I
>> > started
>> > > complaining about the plugin divergence that occurred due to the
move
>> to
>> > > bro/plugins (the right move at the time), but Metron's use of a
local
>> > > directory that hadn't been kept up to date. My current efforts are
an
>> > > attempt to make sure this doesn't happen again, and to take
advantage
>> of
>> > > the bro-pkg benefits.
>> > >
>> > > - The gap between ~3/31 and actual progress on 11/12 is completely
on
>> me
>> > -
>> > > I had intentions of doing this work sooner but failed to do so.
>> > >
>> > > - You can most definitely still install/use the bro plugin without
>> using
>> > > bro-pkg. In fact, the README in my PR still has the instructions on
>> how
>> > to
>> > > do so.
>> > >
>> > >
>> > >
>> > > Q1: The simple explanation is that the only thing that makes a
plugin
>> a
>> > bro
>> > > package is the inclusion of a bro-pkg.meta file, and it includes a
>> > > build_command which could easily be manually performed to install by
>> hand
>> > > (assuming dependencies are met).
>> > >
>> > >
>> > >
>> > > I've worked with other projects that use submodules and while I'm
fine
>> > > discussing it, I suggest that we don't implement it. I put together
a
>> > quick
>> > > example of why here[1], using the bro project as an example since
it's
>> > top
>> > > of mind.
>> > >
>> > >
>> > >
>> > > Q2: I think the answer to Q1 answers this. There is absolutely
nothing
>> > > stopping a git clone && cd $dir && configure && make && make
install,
>> but
>> > > using bro-pkg to install/load takes into account dependencies and
unit
>> > > tests when it is loaded (and thus fails early and more intuitively).
>> It
>> > > only must be a separate repo (or, more technically correct, a git
>> branch
>> > > that includes only the package) because of how bro-pkg works. If
you'd
>> > like
>> > > to get an idea of how this would work in application for Bro users,
>> you
>> > can
>> > > see my test instructions here (specifically step #3). If a 0.1 tag
>> gets
>> > > pushed to apache/metron-bro-plugin-kafka, the command could be
>> `bro-pkg
>> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
>> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the
>> --force is
>> > > just to remove user interaction, for an ansible spin-up).
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > 1: To clone the Bro git repo, you must run `git clone --recursive
>> > > https://github.com/bro/bro` <https://github.com/bro/bro> <
>> https://github.com/bro/bro> (note the
>> > > --recursive). Not too big of a deal,
>> > > but requires that you remember it and existing instructions/blog
posts
>> > may
>> > > give users inaccurate steps. Let's make this worse and try to
checkout
>> > > their latest release, v2.5.2, and automatically update the
submodules
>> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
>> fails
>> > > because aux/plugins (https://github.com/bro/plugins) was removed
>> since
>> > > their latest release. Okay, we can work around this using `git
>> checkout
>> > > v2.5.2` and then remember to `git submodule update` every time you
>> > checkout
>> > > a release or branch. But because they have nested submodules, we
>> actually
>> > > need to run `git submodule update --recursive`. I can't imagine
opting
>> > into
>> > > a workflow anything like this. There are other options as well, such
>> as
>> > git
>> > > subtrees, but those I am less familiar with.
>> > >
>> > >
>> > >
>> > > Jon
>> > >
>> > >
>> > >
>> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>

>> > > wrote:
>> > >
>> > > I am not sure that our use of the plugin necessarily equates to it
>> being
>> > > implicitly coupled to Metron. It seems like the Right Thing To Do,
>> esp.
>> > > for an Apache project would be to make this available for use by the
>> > > greater bro community.
>> > > Unless we expect to do extensive iterative work on the plugin, which
>> > would
>> > > then make the decision to spin it out now premature.
>> > >
>> > > Then again, I might be wrong ;)
>> > >
>> > >
>> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org)
>> wrote:
>> > >
>> > > [Please pardon me that the below is a little labored. I’m trying to
>> > > understand the implications for both release and use, which requires
>> some
>> > > explanation as well as the two questions needed. Q1 and Q2 below are
>> > > probably the same question, asked in slightly different contexts.
>> Please
>> > > consider them together.]
>> > >
>> > > So this made me go back and look at the history that caused us to
put
>> the
>> > > bro plugin in a separate repo. As best I can see, this was in
>> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
>> email
>> > > discussion thread. Also please see
>> > > https://issues.apache.org/jira/browse/METRON-883 for background on
>> the
>> > > plugin itself.
>> > >
>> > > As best I can assemble the many bits brought up in the threads, the
>> > reasons
>> > > to put it in a separate repo was:
>> > > - The plugin was thought to be useful to multiple clients of bro and
>> > kafka,
>> > > including Storm and Spark, as well as Metron.
>> > > - Originally the bro project was maintaining bro plugins and it was
>> > thought
>> > > they might adopt this one.
>> > > - Bro then formalized their plugin framework BUT dumped all plugins
>> out
>> > of
>> > > their sphere of maintenance.
>> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism
>> requires
>> > > that a package live within its own repo”. Jon said “the bro packages
>> > model
>> > > doesn't allow colocation with anything else.”
>> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
>> repo
>> > > was created a few days later. But Metron wasn’t actually modified to
>> > remove
>> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using
the
>> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two
weeks
>> > ago!
>> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
>> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
>> repo
>> > > remain valid, if the bro plugin mechanism is used. But obviously
there
>> > are
>> > > (non-conforming) ways to build the plugin as part of metron, and
>> install
>> > it
>> > > in a way that works.
>> > >
>> > > Q1. I think that last statement needs some explanation. Nick or Jon,
>> can
>> > > you please expand on it, especially wrt how the end user installs
the
>> > > plugin once the plugin is built the two different ways? And whether
>> it’s
>> > > still valuable to have a separate repo for the plugin?
>> > >
>> > > Nick suggests using a submodule approach to managing the bro plugin,
>> for
>> > > Metron versioning purposes. As I understand it, this would continue
>> the
>> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
>> > metron
>> > > code tree for building, versioning, and release purposes. Git
>> submodules
>> > > are documented here:
>> https://git-scm.com/book/en/v2/Git-Tools-Submodules
>> > .
>> > > We would use the submodule capability to clone the
>> > metron-bro-plugin-kafka
>> > > source code into a subdirectory of Metron at the time one clones the
>> > metron
>> > > repo. It would then be released with Metron as part of the source
code
>> > > release for a given version of Metron. Part of the way submodules
are
>> > > managed, is that git stores the SHA1 hash of the submodule into a
file
>> > > named .gitmodules, which in turn gets saved when you do a git push.
So
>> > > indeed submodules would ensure that everyone cloning a given version
>> of
>> > > metron would get the expected “version” (sha, actually) of
>> > > metron-bro-plugin-kafka.
>> > >
>> > > This sounds like a good idea, although it isn’t without cost.
>> Submodules
>> > > impose the need for additional commands to actually get a copy of
the
>> > > submodule source, and if the plugin repo advanced beyond the version
>> in a
>> > > metron repo, it causes some ‘git status’ artifacts that could be
>> > confusing
>> > > to folks who aren’t familiar with submodules. But these can be
>> > documented.
>> > >
>> > > Q2. Nick, what I’m not clear about is the process by which the
>> > > metron-bro-plugin-kafka would be built and “plugged in” by (a)
metron
>> > > developers, and (b) end users. If it “must” be in a separate repo to
>> be
>> > > successfully built and managed by the bro plugin mechanism, does
that
>> > mean
>> > > it can’t be built from the copy in the Metron source tree? Yet until
>> > > November, that’s exactly what we were doing. Do we go back to doing
>> that?
>> > > What does that mean wrt users installing the plugin?
>> > >
>> > > Thanks for your patience in reading this far.
>> > > --Matt
>> > >
>> > >
>> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
>> > >
>> > > I agree with Nick. Since the plugin is tightly coupled with Metron
why
>> > not
>> > > just pull it into the main repo and version it with the rest of the
>> code?
>> > > Do we really need the second repo for the plug-in?
>> > >
>> > > Thanks,
>> > > James
>> > >
>> > >
>> > >
>> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
>> > > >> I would suggest that we institute a release procedure for the
>> package
>> > > >> itself, but I don't think it necessarily has to line up with
metron
>> > > >> releases (happy to be persuaded otherwise). Then we can just link
>> > metron
>> > > >> to metron-bro-plugin-kafka by pointing to specific
>> > > >> metron-bro-plugin-kafka releases (git tags
>> > > >> <
>> > > http://bro-package-manager.readthedocs.io/en/stable/
>> > package.html#package-
>> > > >> versioning>
>> > > >> ).
>> > > >> Right now, full-dev spins up against the
>> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
>> idea
>> > > to
>> > > >> have in place for an upcoming release. That is the crux of why I
>> think
>> > > we
>> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
>> before
>> > > our
>> > > >> next release (working on it as we speak).
>> > > >> Jon
>> > > >
>> > > > ​I replayed Jon's comments from the other thread above.​
>> > > >
>> > > > My initial thought, is that I would not want to manage two
separate
>> > > release
>> > > > processes. I don't want to have a roll call, cut release
candidates
>> and
>> > > > test both.
>> > > >
>> > > > I was thinking we would just need to change some of the
>> > behind-the-scenes
>> > > > processes handled by the release manager. This is one area where I
>> had
>> > > > thought using a submodule in Git would help.
>> > > >
>> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
>> > wrote:
>> > > >
>> > > >> + Restarting the thread to include mentors.
>> > > >>
>> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
>> > external
>> > > >> repository that we set up a while back.
>> > > >>
>> > > >> - Metron Core: git://git.apache.org/metron.git
>> > > >> - Kafka Plugin for Bro: git://git.apache.org/
>> > > >> metron-bro-plugin-kafka.git
>> > > >>
>> > > >> (Q) Do we need to change anything in the release procedure to
>> account
>> > > for
>> > > >> this?
>> > >
>> > > -------------------
>> > > Thank you,
>> > >
>> > > James Sirota
>> > > PMC- Apache Metron
>> > > jsirota AT apache DOT org
>> > >
>> > > --
>> > >
>> > > Jon
>> > >
>> > --
>> >
>> > Jon
>> >
>>
>>
>>
>> --
>
> Jon
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

["Zeolla@GMail.com" <ze...@gmail.com>]

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM Zeolla@GMail.com <ze...@gmail.com> wrote:

> Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release.  Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen <ni...@nickallen.org>
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" <de...@metron.apache.org>
>> Cc: Matt Foley <ma...@apache.org>
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>>
>>
>>
>> I am more interested in getting a release cut.  If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com>
>> wrote:
>>
>> Do we have any further discussion on this?  Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can get
>> the release bus moving again, and right now it seems like (a) is gathering
>> more explicit support.  Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com>
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation.  For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this.  It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron.  This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > >     - open a jira,
>> > >
>> > >     - add the submodule to the Metron code tree,
>> > >
>> > >     - possibly (optionally) add build mechanism to the maven poms, and
>> > >
>> > >     - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen <ni...@nickallen.org>
>> > > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
>> > > Date: Tuesday, November 28, 2017 at 9:09 AM
>> > > To: "dev@metron.apache.org" <de...@metron.apache.org>
>> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
>> > Bro'
>> > >
>> > >
>> > >
>> > > I'll add a bit to Jon's technical comments.
>> > >
>> > >
>> > >
>> > > * We only created a separate repo because it was a technical
>> requirement
>> > to
>> > > leverage the bro-pkg mechanism.
>> > >
>> > > * Leveraging the new bro-pkg mechanism has many advantages as
>> outlined by
>> > > Jon.
>> > >
>> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
>> install
>> > the
>> > > plugin exactly how we use to.
>> > >
>> > >
>> > >
>> > > While I agree with Jon's technical comments, I disagree with the
>> > > non-technical ones.
>> > >
>> > >
>> > >
>> > > (1) I do not want to change our release management process. While we
>> > needed
>> > > to make a new repo (a technical change), I did not want that to change
>> > how
>> > > we operate as a community (our procedures, policies, versioning and
>> > release
>> > > cycles).
>> > >
>> > >
>> > >
>> > > (2) I see no value in adopting a separate release management process
>> for
>> > > the Bro plugin alone. Having a separate release process does not make
>> the
>> > > plugin *more* available to the Bro community.
>> > >
>> > >
>> > >
>> > > (3) I also see no value in positioning the plugin to be spun-out of
>> the
>> > > Metron project. It is a part of Metron and I want to see it benefit
>> and
>> > > evolve "the Apache-way".
>> > >
>> > >
>> > >
>> > > In my mind, the best way to accommodate the additional repo, while
>> > > minimizing changes to our release management process, is to treat the
>> new
>> > > repo as a submodule. I fail to see significant downsides to this
>> > approach.
>> > > A few extract command-line options do not seem overly onerous to me.
>> > >
>> > >
>> > >
>> > > Many thanks go to Jon for all the hard work he has put in enhancing
>> the
>> > > plugin.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
>> > > wrote:
>> > >
>> > > In an attempt to keep this from becoming unbearably long, I will try
>> to
>> > > keep my responses short, but I would be happy to elaborate. That's a
>> > fairly
>> > > good timeline and summary, but here are some clarifications in
>> > > corresponding order:
>> > >
>> > >
>> > >
>> > > - The plugin history is quite short and you can probably get a good
>> bit
>> > of
>> > > context just by looking at the commits.
>> > >
>> > > - The plugin is only useful to the bro community, but it is rather
>> > popular.
>> > >
>> > > - The Bro team created the idea of bro packages, which can include bro
>> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
>> > > 'plugins' repo, they moved to have a 'packages' repo which is by
>> default
>> > > referenced by a bro-pkg tool they wrote for package management.
>> > >
>> > > - I believe I kicked this off (or at least I did in my head) when I
>> > started
>> > > complaining about the plugin divergence that occurred due to the move
>> to
>> > > bro/plugins (the right move at the time), but Metron's use of a local
>> > > directory that hadn't been kept up to date. My current efforts are an
>> > > attempt to make sure this doesn't happen again, and to take advantage
>> of
>> > > the bro-pkg benefits.
>> > >
>> > > - The gap between ~3/31 and actual progress on 11/12 is completely on
>> me
>> > -
>> > > I had intentions of doing this work sooner but failed to do so.
>> > >
>> > > - You can most definitely still install/use the bro plugin without
>> using
>> > > bro-pkg. In fact, the README in my PR still has the instructions on
>> how
>> > to
>> > > do so.
>> > >
>> > >
>> > >
>> > > Q1: The simple explanation is that the only thing that makes a plugin
>> a
>> > bro
>> > > package is the inclusion of a bro-pkg.meta file, and it includes a
>> > > build_command which could easily be manually performed to install by
>> hand
>> > > (assuming dependencies are met).
>> > >
>> > >
>> > >
>> > > I've worked with other projects that use submodules and while I'm fine
>> > > discussing it, I suggest that we don't implement it. I put together a
>> > quick
>> > > example of why here[1], using the bro project as an example since it's
>> > top
>> > > of mind.
>> > >
>> > >
>> > >
>> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
>> > > stopping a git clone && cd $dir && configure && make && make install,
>> but
>> > > using bro-pkg to install/load takes into account dependencies and unit
>> > > tests when it is loaded (and thus fails early and more intuitively).
>> It
>> > > only must be a separate repo (or, more technically correct, a git
>> branch
>> > > that includes only the package) because of how bro-pkg works. If you'd
>> > like
>> > > to get an idea of how this would work in application for Bro users,
>> you
>> > can
>> > > see my test instructions here (specifically step #3). If a 0.1 tag
>> gets
>> > > pushed to apache/metron-bro-plugin-kafka, the command could be
>> `bro-pkg
>> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
>> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the
>> --force is
>> > > just to remove user interaction, for an ansible spin-up).
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > 1: To clone the Bro git repo, you must run `git clone --recursive
>> > > https://github.com/bro/bro` <https://github.com/bro/bro> <
>> https://github.com/bro/bro> (note the
>> > > --recursive). Not too big of a deal,
>> > > but requires that you remember it and existing instructions/blog posts
>> > may
>> > > give users inaccurate steps. Let's make this worse and try to checkout
>> > > their latest release, v2.5.2, and automatically update the submodules
>> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
>> fails
>> > > because aux/plugins (https://github.com/bro/plugins) was removed
>> since
>> > > their latest release. Okay, we can work around this using `git
>> checkout
>> > > v2.5.2` and then remember to `git submodule update` every time you
>> > checkout
>> > > a release or branch. But because they have nested submodules, we
>> actually
>> > > need to run `git submodule update --recursive`. I can't imagine opting
>> > into
>> > > a workflow anything like this. There are other options as well, such
>> as
>> > git
>> > > subtrees, but those I am less familiar with.
>> > >
>> > >
>> > >
>> > > Jon
>> > >
>> > >
>> > >
>> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
>> > > wrote:
>> > >
>> > > I am not sure that our use of the plugin necessarily equates to it
>> being
>> > > implicitly coupled to Metron. It seems like the Right Thing To Do,
>> esp.
>> > > for an Apache project would be to make this available for use by the
>> > > greater bro community.
>> > > Unless we expect to do extensive iterative work on the plugin, which
>> > would
>> > > then make the decision to spin it out now premature.
>> > >
>> > > Then again, I might be wrong ;)
>> > >
>> > >
>> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org)
>> wrote:
>> > >
>> > > [Please pardon me that the below is a little labored. I’m trying to
>> > > understand the implications for both release and use, which requires
>> some
>> > > explanation as well as the two questions needed. Q1 and Q2 below are
>> > > probably the same question, asked in slightly different contexts.
>> Please
>> > > consider them together.]
>> > >
>> > > So this made me go back and look at the history that caused us to put
>> the
>> > > bro plugin in a separate repo. As best I can see, this was in
>> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
>> email
>> > > discussion thread. Also please see
>> > > https://issues.apache.org/jira/browse/METRON-883 for background on
>> the
>> > > plugin itself.
>> > >
>> > > As best I can assemble the many bits brought up in the threads, the
>> > reasons
>> > > to put it in a separate repo was:
>> > > - The plugin was thought to be useful to multiple clients of bro and
>> > kafka,
>> > > including Storm and Spark, as well as Metron.
>> > > - Originally the bro project was maintaining bro plugins and it was
>> > thought
>> > > they might adopt this one.
>> > > - Bro then formalized their plugin framework BUT dumped all plugins
>> out
>> > of
>> > > their sphere of maintenance.
>> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism
>> requires
>> > > that a package live within its own repo”. Jon said “the bro packages
>> > model
>> > > doesn't allow colocation with anything else.”
>> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
>> repo
>> > > was created a few days later. But Metron wasn’t actually modified to
>> > remove
>> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
>> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
>> > ago!
>> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
>> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
>> repo
>> > > remain valid, if the bro plugin mechanism is used. But obviously there
>> > are
>> > > (non-conforming) ways to build the plugin as part of metron, and
>> install
>> > it
>> > > in a way that works.
>> > >
>> > > Q1. I think that last statement needs some explanation. Nick or Jon,
>> can
>> > > you please expand on it, especially wrt how the end user installs the
>> > > plugin once the plugin is built the two different ways? And whether
>> it’s
>> > > still valuable to have a separate repo for the plugin?
>> > >
>> > > Nick suggests using a submodule approach to managing the bro plugin,
>> for
>> > > Metron versioning purposes. As I understand it, this would continue
>> the
>> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
>> > metron
>> > > code tree for building, versioning, and release purposes. Git
>> submodules
>> > > are documented here:
>> https://git-scm.com/book/en/v2/Git-Tools-Submodules
>> > .
>> > > We would use the submodule capability to clone the
>> > metron-bro-plugin-kafka
>> > > source code into a subdirectory of Metron at the time one clones the
>> > metron
>> > > repo. It would then be released with Metron as part of the source code
>> > > release for a given version of Metron. Part of the way submodules are
>> > > managed, is that git stores the SHA1 hash of the submodule into a file
>> > > named .gitmodules, which in turn gets saved when you do a git push. So
>> > > indeed submodules would ensure that everyone cloning a given version
>> of
>> > > metron would get the expected “version” (sha, actually) of
>> > > metron-bro-plugin-kafka.
>> > >
>> > > This sounds like a good idea, although it isn’t without cost.
>> Submodules
>> > > impose the need for additional commands to actually get a copy of the
>> > > submodule source, and if the plugin repo advanced beyond the version
>> in a
>> > > metron repo, it causes some ‘git status’ artifacts that could be
>> > confusing
>> > > to folks who aren’t familiar with submodules. But these can be
>> > documented.
>> > >
>> > > Q2. Nick, what I’m not clear about is the process by which the
>> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
>> > > developers, and (b) end users. If it “must” be in a separate repo to
>> be
>> > > successfully built and managed by the bro plugin mechanism, does that
>> > mean
>> > > it can’t be built from the copy in the Metron source tree? Yet until
>> > > November, that’s exactly what we were doing. Do we go back to doing
>> that?
>> > > What does that mean wrt users installing the plugin?
>> > >
>> > > Thanks for your patience in reading this far.
>> > > --Matt
>> > >
>> > >
>> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
>> > >
>> > > I agree with Nick. Since the plugin is tightly coupled with Metron why
>> > not
>> > > just pull it into the main repo and version it with the rest of the
>> code?
>> > > Do we really need the second repo for the plug-in?
>> > >
>> > > Thanks,
>> > > James
>> > >
>> > >
>> > >
>> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
>> > > >> I would suggest that we institute a release procedure for the
>> package
>> > > >> itself, but I don't think it necessarily has to line up with metron
>> > > >> releases (happy to be persuaded otherwise). Then we can just link
>> > metron
>> > > >> to metron-bro-plugin-kafka by pointing to specific
>> > > >> metron-bro-plugin-kafka releases (git tags
>> > > >> <
>> > > http://bro-package-manager.readthedocs.io/en/stable/
>> > package.html#package-
>> > > >> versioning>
>> > > >> ).
>> > > >> Right now, full-dev spins up against the
>> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
>> idea
>> > > to
>> > > >> have in place for an upcoming release. That is the crux of why I
>> think
>> > > we
>> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
>> before
>> > > our
>> > > >> next release (working on it as we speak).
>> > > >> Jon
>> > > >
>> > > > ​I replayed Jon's comments from the other thread above.​
>> > > >
>> > > > My initial thought, is that I would not want to manage two separate
>> > > release
>> > > > processes. I don't want to have a roll call, cut release candidates
>> and
>> > > > test both.
>> > > >
>> > > > I was thinking we would just need to change some of the
>> > behind-the-scenes
>> > > > processes handled by the release manager. This is one area where I
>> had
>> > > > thought using a submodule in Git would help.
>> > > >
>> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
>> > wrote:
>> > > >
>> > > >> + Restarting the thread to include mentors.
>> > > >>
>> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
>> > external
>> > > >> repository that we set up a while back.
>> > > >>
>> > > >> - Metron Core: git://git.apache.org/metron.git
>> > > >> - Kafka Plugin for Bro: git://git.apache.org/
>> > > >> metron-bro-plugin-kafka.git
>> > > >>
>> > > >> (Q) Do we need to change anything in the release procedure to
>> account
>> > > for
>> > > >> this?
>> > >
>> > > -------------------
>> > > Thank you,
>> > >
>> > > James Sirota
>> > > PMC- Apache Metron
>> > > jsirota AT apache DOT org
>> > >
>> > > --
>> > >
>> > > Jon
>> > >
>> > --
>> >
>> > Jon
>> >
>>
>>
>>
>> --
>
> Jon
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

["Zeolla@GMail.com" <ze...@gmail.com>]

Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone

Jon

On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote:

> I can start the release process tonight.
>
>
>
> Jon, you mentioned you want to commit
>
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> before the release.  Is it convenient for you to do so today?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen <ni...@nickallen.org>
> Date: Thursday, December 7, 2017 at 10:13 AM
> To: "dev@metron.apache.org" <de...@metron.apache.org>
> Cc: Matt Foley <ma...@apache.org>
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I am more interested in getting a release cut.  If me moving to the (a)
> camp gets us to consensus and cuts a release faster, then I'll do it.
> Let's get this release train moving.
>
>
>
> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com>
> wrote:
>
> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
>
> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > >     - open a jira,
> > >
> > >     - add the submodule to the Metron code tree,
> > >
> > >     - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > >     - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen <ni...@nickallen.org>
> > > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" <de...@metron.apache.org>
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technical ones.
> > >
> > >
> > >
> > > (1) I do not want to change our release management process. While we
> > needed
> > > to make a new repo (a technical change), I did not want that to change
> > how
> > > we operate as a community (our procedures, policies, versioning and
> > release
> > > cycles).
> > >
> > >
> > >
> > > (2) I see no value in adopting a separate release management process
> for
> > > the Bro plugin alone. Having a separate release process does not make
> the
> > > plugin *more* available to the Bro community.
> > >
> > >
> > >
> > > (3) I also see no value in positioning the plugin to be spun-out of the
> > > Metron project. It is a part of Metron and I want to see it benefit and
> > > evolve "the Apache-way".
> > >
> > >
> > >
> > > In my mind, the best way to accommodate the additional repo, while
> > > minimizing changes to our release management process, is to treat the
> new
> > > repo as a submodule. I fail to see significant downsides to this
> > approach.
> > > A few extract command-line options do not seem overly onerous to me.
> > >
> > >
> > >
> > > Many thanks go to Jon for all the hard work he has put in enhancing the
> > > plugin.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > > In an attempt to keep this from becoming unbearably long, I will try to
> > > keep my responses short, but I would be happy to elaborate. That's a
> > fairly
> > > good timeline and summary, but here are some clarifications in
> > > corresponding order:
> > >
> > >
> > >
> > > - The plugin history is quite short and you can probably get a good bit
> > of
> > > context just by looking at the commits.
> > >
> > > - The plugin is only useful to the bro community, but it is rather
> > popular.
> > >
> > > - The Bro team created the idea of bro packages, which can include bro
> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > > 'plugins' repo, they moved to have a 'packages' repo which is by
> default
> > > referenced by a bro-pkg tool they wrote for package management.
> > >
> > > - I believe I kicked this off (or at least I did in my head) when I
> > started
> > > complaining about the plugin divergence that occurred due to the move
> to
> > > bro/plugins (the right move at the time), but Metron's use of a local
> > > directory that hadn't been kept up to date. My current efforts are an
> > > attempt to make sure this doesn't happen again, and to take advantage
> of
> > > the bro-pkg benefits.
> > >
> > > - The gap between ~3/31 and actual progress on 11/12 is completely on
> me
> > -
> > > I had intentions of doing this work sooner but failed to do so.
> > >
> > > - You can most definitely still install/use the bro plugin without
> using
> > > bro-pkg. In fact, the README in my PR still has the instructions on how
> > to
> > > do so.
> > >
> > >
> > >
> > > Q1: The simple explanation is that the only thing that makes a plugin a
> > bro
> > > package is the inclusion of a bro-pkg.meta file, and it includes a
> > > build_command which could easily be manually performed to install by
> hand
> > > (assuming dependencies are met).
> > >
> > >
> > >
> > > I've worked with other projects that use submodules and while I'm fine
> > > discussing it, I suggest that we don't implement it. I put together a
> > quick
> > > example of why here[1], using the bro project as an example since it's
> > top
> > > of mind.
> > >
> > >
> > >
> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
> > > stopping a git clone && cd $dir && configure && make && make install,
> but
> > > using bro-pkg to install/load takes into account dependencies and unit
> > > tests when it is loaded (and thus fails early and more intuitively). It
> > > only must be a separate repo (or, more technically correct, a git
> branch
> > > that includes only the package) because of how bro-pkg works. If you'd
> > like
> > > to get an idea of how this would work in application for Bro users, you
> > can
> > > see my test instructions here (specifically step #3). If a 0.1 tag gets
> > > pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force
> is
> > > just to remove user interaction, for an ansible spin-up).
> > >
> > >
> > >
> > >
> > >
> > > 1: To clone the Bro git repo, you must run `git clone --recursive
> > > https://github.com/bro/bro` <https://github.com/bro/bro> <
> https://github.com/bro/bro> (note the
> > > --recursive). Not too big of a deal,
> > > but requires that you remember it and existing instructions/blog posts
> > may
> > > give users inaccurate steps. Let's make this worse and try to checkout
> > > their latest release, v2.5.2, and automatically update the submodules
> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
> fails
> > > because aux/plugins (https://github.com/bro/plugins) was removed since
> > > their latest release. Okay, we can work around this using `git checkout
> > > v2.5.2` and then remember to `git submodule update` every time you
> > checkout
> > > a release or branch. But because they have nested submodules, we
> actually
> > > need to run `git submodule update --recursive`. I can't imagine opting
> > into
> > > a workflow anything like this. There are other options as well, such as
> > git
> > > subtrees, but those I am less familiar with.
> > >
> > >
> > >
> > > Jon
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > >
> > > I am not sure that our use of the plugin necessarily equates to it
> being
> > > implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
> > > for an Apache project would be to make this available for use by the
> > > greater bro community.
> > > Unless we expect to do extensive iterative work on the plugin, which
> > would
> > > then make the decision to spin it out now premature.
> > >
> > > Then again, I might be wrong ;)
> > >
> > >
> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
> > >
> > > [Please pardon me that the below is a little labored. I’m trying to
> > > understand the implications for both release and use, which requires
> some
> > > explanation as well as the two questions needed. Q1 and Q2 below are
> > > probably the same question, asked in slightly different contexts.
> Please
> > > consider them together.]
> > >
> > > So this made me go back and look at the history that caused us to put
> the
> > > bro plugin in a separate repo. As best I can see, this was in
> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
> email
> > > discussion thread. Also please see
> > > https://issues.apache.org/jira/browse/METRON-883 for background on the
> > > plugin itself.
> > >
> > > As best I can assemble the many bits brought up in the threads, the
> > reasons
> > > to put it in a separate repo was:
> > > - The plugin was thought to be useful to multiple clients of bro and
> > kafka,
> > > including Storm and Spark, as well as Metron.
> > > - Originally the bro project was maintaining bro plugins and it was
> > thought
> > > they might adopt this one.
> > > - Bro then formalized their plugin framework BUT dumped all plugins out
> > of
> > > their sphere of maintenance.
> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
> > > that a package live within its own repo”. Jon said “the bro packages
> > model
> > > doesn't allow colocation with anything else.”
> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
> repo
> > > was created a few days later. But Metron wasn’t actually modified to
> > remove
> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
> > ago!
> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
> repo
> > > remain valid, if the bro plugin mechanism is used. But obviously there
> > are
> > > (non-conforming) ways to build the plugin as part of metron, and
> install
> > it
> > > in a way that works.
> > >
> > > Q1. I think that last statement needs some explanation. Nick or Jon,
> can
> > > you please expand on it, especially wrt how the end user installs the
> > > plugin once the plugin is built the two different ways? And whether
> it’s
> > > still valuable to have a separate repo for the plugin?
> > >
> > > Nick suggests using a submodule approach to managing the bro plugin,
> for
> > > Metron versioning purposes. As I understand it, this would continue the
> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
> > metron
> > > code tree for building, versioning, and release purposes. Git
> submodules
> > > are documented here:
> https://git-scm.com/book/en/v2/Git-Tools-Submodules
> > .
> > > We would use the submodule capability to clone the
> > metron-bro-plugin-kafka
> > > source code into a subdirectory of Metron at the time one clones the
> > metron
> > > repo. It would then be released with Metron as part of the source code
> > > release for a given version of Metron. Part of the way submodules are
> > > managed, is that git stores the SHA1 hash of the submodule into a file
> > > named .gitmodules, which in turn gets saved when you do a git push. So
> > > indeed submodules would ensure that everyone cloning a given version of
> > > metron would get the expected “version” (sha, actually) of
> > > metron-bro-plugin-kafka.
> > >
> > > This sounds like a good idea, although it isn’t without cost.
> Submodules
> > > impose the need for additional commands to actually get a copy of the
> > > submodule source, and if the plugin repo advanced beyond the version
> in a
> > > metron repo, it causes some ‘git status’ artifacts that could be
> > confusing
> > > to folks who aren’t familiar with submodules. But these can be
> > documented.
> > >
> > > Q2. Nick, what I’m not clear about is the process by which the
> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
> > > developers, and (b) end users. If it “must” be in a separate repo to be
> > > successfully built and managed by the bro plugin mechanism, does that
> > mean
> > > it can’t be built from the copy in the Metron source tree? Yet until
> > > November, that’s exactly what we were doing. Do we go back to doing
> that?
> > > What does that mean wrt users installing the plugin?
> > >
> > > Thanks for your patience in reading this far.
> > > --Matt
> > >
> > >
> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
> > >
> > > I agree with Nick. Since the plugin is tightly coupled with Metron why
> > not
> > > just pull it into the main repo and version it with the rest of the
> code?
> > > Do we really need the second repo for the plug-in?
> > >
> > > Thanks,
> > > James
> > >
> > >
> > >
> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
> > > >> I would suggest that we institute a release procedure for the
> package
> > > >> itself, but I don't think it necessarily has to line up with metron
> > > >> releases (happy to be persuaded otherwise). Then we can just link
> > metron
> > > >> to metron-bro-plugin-kafka by pointing to specific
> > > >> metron-bro-plugin-kafka releases (git tags
> > > >> <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > package.html#package-
> > > >> versioning>
> > > >> ).
> > > >> Right now, full-dev spins up against the
> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
> idea
> > > to
> > > >> have in place for an upcoming release. That is the crux of why I
> think
> > > we
> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
> before
> > > our
> > > >> next release (working on it as we speak).
> > > >> Jon
> > > >
> > > > ​I replayed Jon's comments from the other thread above.​
> > > >
> > > > My initial thought, is that I would not want to manage two separate
> > > release
> > > > processes. I don't want to have a roll call, cut release candidates
> and
> > > > test both.
> > > >
> > > > I was thinking we would just need to change some of the
> > behind-the-scenes
> > > > processes handled by the release manager. This is one area where I
> had
> > > > thought using a submodule in Git would help.
> > > >
> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> > > >
> > > >> + Restarting the thread to include mentors.
> > > >>
> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
> > external
> > > >> repository that we set up a while back.
> > > >>
> > > >> - Metron Core: git://git.apache.org/metron.git
> > > >> - Kafka Plugin for Bro: git://git.apache.org/
> > > >> metron-bro-plugin-kafka.git
> > > >>
> > > >> (Q) Do we need to change anything in the release procedure to
> account
> > > for
> > > >> this?
> > >
> > > -------------------
> > > Thank you,
> > >
> > > James Sirota
> > > PMC- Apache Metron
> > > jsirota AT apache DOT org
> > >
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>
>
>
> --

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley <ma...@apache.org>]

I can start the release process tonight.

 

Jon, you mentioned you want to commit 

> https://github.com/apache/metron/pull/847 and
> https://github.com/apache/metron-bro-plugin-kafka/pull/4

before the release.  Is it convenient for you to do so today?

 

Thanks,

--Matt

 

From: Nick Allen <ni...@nickallen.org>
Date: Thursday, December 7, 2017 at 10:13 AM
To: "dev@metron.apache.org" <de...@metron.apache.org>
Cc: Matt Foley <ma...@apache.org>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

 

I am more interested in getting a release cut.  If me moving to the (a) camp gets us to consensus and cuts a release faster, then I'll do it.  Let's get this release train moving.

 

On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com> wrote:

Do we have any further discussion on this?  Pardon me if I misstate
anyone's position, but it seems like we have a couple people (Otto and Jon
and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
section of people like myself without a particular horse in the race.

It seems like we need to come to some sort of consensus so that we can get
the release bus moving again, and right now it seems like (a) is gathering
more explicit support.  Do we have a compelling reason to not do (a)? To be
honest, my main worry is more "If we do (a) are we going to be miserable if
we need to iterate or adjust?" I'm not seeing anything that suggests
anything too terrible, so unless we see some more discussion, I suggest we
move forward with (a).


On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:

> I would prefer a, but I was initially thinking of doing the plugin first
> and then get in the two PRs out to use this new tag, which are already +1'd
> and just waiting on this conversation.  For reference,
> https://github.com/apache/metron/pull/847 and
> https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> Jon
>
> On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
>
> > It seems to me, as I believe I have stated before that a) feels like the
> > proper way to handle this.  It is how I have seen other projects like
> NiFi
> > handle things as well.
> >
> >
> >
> > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
> >
> > Okay, looking at this from the perspective of making a release:
> >
> >
> >
> > We have two choices:
> >
> > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > metron-bro-plugin-kafka, at the same time and using the same process
> > (modulo the necessary) as Metron.  This is dirt simple.
> >
> > b) I or someone needs to:
> >
> >     - open a jira,
> >
> >     - add the submodule to the Metron code tree,
> >
> >     - possibly (optionally) add build mechanism to the maven poms, and
> >
> >     - document as much as we think appropriate regarding what it is, how
> to
> > build it, and how to update it,
> >
> > and commit that before the 0.4.2 release.
> >
> >
> >
> > What is the will of the community?
> >
> > Thanks,
> >
> > --Matt
> >
> >
> >
> > From: Nick Allen <ni...@nickallen.org>
> > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
> > Date: Tuesday, November 28, 2017 at 9:09 AM
> > To: "dev@metron.apache.org" <de...@metron.apache.org>
> > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> Bro'
> >
> >
> >
> > I'll add a bit to Jon's technical comments.
> >
> >
> >
> > * We only created a separate repo because it was a technical requirement
> to
> > leverage the bro-pkg mechanism.
> >
> > * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> > Jon.
> >
> > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> the
> > plugin exactly how we use to.
> >
> >
> >
> > While I agree with Jon's technical comments, I disagree with the
> > non-technical ones.
> >
> >
> >
> > (1) I do not want to change our release management process. While we
> needed
> > to make a new repo (a technical change), I did not want that to change
> how
> > we operate as a community (our procedures, policies, versioning and
> release
> > cycles).
> >
> >
> >
> > (2) I see no value in adopting a separate release management process for
> > the Bro plugin alone. Having a separate release process does not make the
> > plugin *more* available to the Bro community.
> >
> >
> >
> > (3) I also see no value in positioning the plugin to be spun-out of the
> > Metron project. It is a part of Metron and I want to see it benefit and
> > evolve "the Apache-way".
> >
> >
> >
> > In my mind, the best way to accommodate the additional repo, while
> > minimizing changes to our release management process, is to treat the new
> > repo as a submodule. I fail to see significant downsides to this
> approach.
> > A few extract command-line options do not seem overly onerous to me.
> >
> >
> >
> > Many thanks go to Jon for all the hard work he has put in enhancing the
> > plugin.
> >
> >
> >
> >
> >
> > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> > In an attempt to keep this from becoming unbearably long, I will try to
> > keep my responses short, but I would be happy to elaborate. That's a
> fairly
> > good timeline and summary, but here are some clarifications in
> > corresponding order:
> >
> >
> >
> > - The plugin history is quite short and you can probably get a good bit
> of
> > context just by looking at the commits.
> >
> > - The plugin is only useful to the bro community, but it is rather
> popular.
> >
> > - The Bro team created the idea of bro packages, which can include bro
> > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > 'plugins' repo, they moved to have a 'packages' repo which is by default
> > referenced by a bro-pkg tool they wrote for package management.
> >
> > - I believe I kicked this off (or at least I did in my head) when I
> started
> > complaining about the plugin divergence that occurred due to the move to
> > bro/plugins (the right move at the time), but Metron's use of a local
> > directory that hadn't been kept up to date. My current efforts are an
> > attempt to make sure this doesn't happen again, and to take advantage of
> > the bro-pkg benefits.
> >
> > - The gap between ~3/31 and actual progress on 11/12 is completely on me
> -
> > I had intentions of doing this work sooner but failed to do so.
> >
> > - You can most definitely still install/use the bro plugin without using
> > bro-pkg. In fact, the README in my PR still has the instructions on how
> to
> > do so.
> >
> >
> >
> > Q1: The simple explanation is that the only thing that makes a plugin a
> bro
> > package is the inclusion of a bro-pkg.meta file, and it includes a
> > build_command which could easily be manually performed to install by hand
> > (assuming dependencies are met).
> >
> >
> >
> > I've worked with other projects that use submodules and while I'm fine
> > discussing it, I suggest that we don't implement it. I put together a
> quick
> > example of why here[1], using the bro project as an example since it's
> top
> > of mind.
> >
> >
> >
> > Q2: I think the answer to Q1 answers this. There is absolutely nothing
> > stopping a git clone && cd $dir && configure && make && make install, but
> > using bro-pkg to install/load takes into account dependencies and unit
> > tests when it is loaded (and thus fails early and more intuitively). It
> > only must be a separate repo (or, more technically correct, a git branch
> > that includes only the package) because of how bro-pkg works. If you'd
> like
> > to get an idea of how this would work in application for Bro users, you
> can
> > see my test instructions here (specifically step #3). If a 0.1 tag gets
> > pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
> > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
> > apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
> > just to remove user interaction, for an ansible spin-up).
> >
> >
> >
> >
> >
> > 1: To clone the Bro git repo, you must run `git clone --recursive
> > https://github.com/bro/bro` <https://github.com/bro/bro> (note the
> > --recursive). Not too big of a deal,
> > but requires that you remember it and existing instructions/blog posts
> may
> > give users inaccurate steps. Let's make this worse and try to checkout
> > their latest release, v2.5.2, and automatically update the submodules
> > appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
> > because aux/plugins (https://github.com/bro/plugins) was removed since
> > their latest release. Okay, we can work around this using `git checkout
> > v2.5.2` and then remember to `git submodule update` every time you
> checkout
> > a release or branch. But because they have nested submodules, we actually
> > need to run `git submodule update --recursive`. I can't imagine opting
> into
> > a workflow anything like this. There are other options as well, such as
> git
> > subtrees, but those I am less familiar with.
> >
> >
> >
> > Jon
> >
> >
> >
> > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > I am not sure that our use of the plugin necessarily equates to it being
> > implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
> > for an Apache project would be to make this available for use by the
> > greater bro community.
> > Unless we expect to do extensive iterative work on the plugin, which
> would
> > then make the decision to spin it out now premature.
> >
> > Then again, I might be wrong ;)
> >
> >
> > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
> >
> > [Please pardon me that the below is a little labored. I’m trying to
> > understand the implications for both release and use, which requires some
> > explanation as well as the two questions needed. Q1 and Q2 below are
> > probably the same question, asked in slightly different contexts. Please
> > consider them together.]
> >
> > So this made me go back and look at the history that caused us to put the
> > bro plugin in a separate repo. As best I can see, this was in
> > https://issues.apache.org/jira/browse/METRON-813 , which cites an email
> > discussion thread. Also please see
> > https://issues.apache.org/jira/browse/METRON-883 for background on the
> > plugin itself.
> >
> > As best I can assemble the many bits brought up in the threads, the
> reasons
> > to put it in a separate repo was:
> > - The plugin was thought to be useful to multiple clients of bro and
> kafka,
> > including Storm and Spark, as well as Metron.
> > - Originally the bro project was maintaining bro plugins and it was
> thought
> > they might adopt this one.
> > - Bro then formalized their plugin framework BUT dumped all plugins out
> of
> > their sphere of maintenance.
> > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
> > that a package live within its own repo”. Jon said “the bro packages
> model
> > doesn't allow colocation with anything else.”
> > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
> > was created a few days later. But Metron wasn’t actually modified to
> remove
> > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
> > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
> ago!
> > – with https://issues.apache.org/jira/browse/METRON-1309 .
> > - Presumably the need to have metron-bro-plugin-kafka in a separate repo
> > remain valid, if the bro plugin mechanism is used. But obviously there
> are
> > (non-conforming) ways to build the plugin as part of metron, and install
> it
> > in a way that works.
> >
> > Q1. I think that last statement needs some explanation. Nick or Jon, can
> > you please expand on it, especially wrt how the end user installs the
> > plugin once the plugin is built the two different ways? And whether it’s
> > still valuable to have a separate repo for the plugin?
> >
> > Nick suggests using a submodule approach to managing the bro plugin, for
> > Metron versioning purposes. As I understand it, this would continue the
> > existence of the metron-bro-plugin-kafka repo, but copy it into the
> metron
> > code tree for building, versioning, and release purposes. Git submodules
> > are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules
> .
> > We would use the submodule capability to clone the
> metron-bro-plugin-kafka
> > source code into a subdirectory of Metron at the time one clones the
> metron
> > repo. It would then be released with Metron as part of the source code
> > release for a given version of Metron. Part of the way submodules are
> > managed, is that git stores the SHA1 hash of the submodule into a file
> > named .gitmodules, which in turn gets saved when you do a git push. So
> > indeed submodules would ensure that everyone cloning a given version of
> > metron would get the expected “version” (sha, actually) of
> > metron-bro-plugin-kafka.
> >
> > This sounds like a good idea, although it isn’t without cost. Submodules
> > impose the need for additional commands to actually get a copy of the
> > submodule source, and if the plugin repo advanced beyond the version in a
> > metron repo, it causes some ‘git status’ artifacts that could be
> confusing
> > to folks who aren’t familiar with submodules. But these can be
> documented.
> >
> > Q2. Nick, what I’m not clear about is the process by which the
> > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
> > developers, and (b) end users. If it “must” be in a separate repo to be
> > successfully built and managed by the bro plugin mechanism, does that
> mean
> > it can’t be built from the copy in the Metron source tree? Yet until
> > November, that’s exactly what we were doing. Do we go back to doing that?
> > What does that mean wrt users installing the plugin?
> >
> > Thanks for your patience in reading this far.
> > --Matt
> >
> >
> > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
> >
> > I agree with Nick. Since the plugin is tightly coupled with Metron why
> not
> > just pull it into the main repo and version it with the rest of the code?
> > Do we really need the second repo for the plug-in?
> >
> > Thanks,
> > James
> >
> >
> >
> > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
> > >> I would suggest that we institute a release procedure for the package
> > >> itself, but I don't think it necessarily has to line up with metron
> > >> releases (happy to be persuaded otherwise). Then we can just link
> metron
> > >> to metron-bro-plugin-kafka by pointing to specific
> > >> metron-bro-plugin-kafka releases (git tags
> > >> <
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > >> versioning>
> > >> ).
> > >> Right now, full-dev spins up against the
> > >> apache/metron-bro-plugin-kafka master branch, which is not a good idea
> > to
> > >> have in place for an upcoming release. That is the crux of why I think
> > we
> > >> need to finalize the move to bro 2.5.2 and the plugin packaging before
> > our
> > >> next release (working on it as we speak).
> > >> Jon
> > >
> > > ​I replayed Jon's comments from the other thread above.​
> > >
> > > My initial thought, is that I would not want to manage two separate
> > release
> > > processes. I don't want to have a roll call, cut release candidates and
> > > test both.
> > >
> > > I was thinking we would just need to change some of the
> behind-the-scenes
> > > processes handled by the release manager. This is one area where I had
> > > thought using a submodule in Git would help.
> > >
> > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
> wrote:
> > >
> > >> + Restarting the thread to include mentors.
> > >>
> > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
> external
> > >> repository that we set up a while back.
> > >>
> > >> - Metron Core: git://git.apache.org/metron.git
> > >> - Kafka Plugin for Bro: git://git.apache.org/
> > >> metron-bro-plugin-kafka.git
> > >>
> > >> (Q) Do we need to change anything in the release procedure to account
> > for
> > >> this?
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
> > --
> >
> > Jon
> >
> --
>
> Jon
>

 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Nick Allen <ni...@nickallen.org>]

I am more interested in getting a release cut.  If me moving to the (a)
camp gets us to consensus and cuts a release faster, then I'll do it.
Let's get this release train moving.

On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com> wrote:

> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > >     - open a jira,
> > >
> > >     - add the submodule to the Metron code tree,
> > >
> > >     - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > >     - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen <ni...@nickallen.org>
> > > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" <de...@metron.apache.org>
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technical ones.
> > >
> > >
> > >
> > > (1) I do not want to change our release management process. While we
> > needed
> > > to make a new repo (a technical change), I did not want that to change
> > how
> > > we operate as a community (our procedures, policies, versioning and
> > release
> > > cycles).
> > >
> > >
> > >
> > > (2) I see no value in adopting a separate release management process
> for
> > > the Bro plugin alone. Having a separate release process does not make
> the
> > > plugin *more* available to the Bro community.
> > >
> > >
> > >
> > > (3) I also see no value in positioning the plugin to be spun-out of the
> > > Metron project. It is a part of Metron and I want to see it benefit and
> > > evolve "the Apache-way".
> > >
> > >
> > >
> > > In my mind, the best way to accommodate the additional repo, while
> > > minimizing changes to our release management process, is to treat the
> new
> > > repo as a submodule. I fail to see significant downsides to this
> > approach.
> > > A few extract command-line options do not seem overly onerous to me.
> > >
> > >
> > >
> > > Many thanks go to Jon for all the hard work he has put in enhancing the
> > > plugin.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > > In an attempt to keep this from becoming unbearably long, I will try to
> > > keep my responses short, but I would be happy to elaborate. That's a
> > fairly
> > > good timeline and summary, but here are some clarifications in
> > > corresponding order:
> > >
> > >
> > >
> > > - The plugin history is quite short and you can probably get a good bit
> > of
> > > context just by looking at the commits.
> > >
> > > - The plugin is only useful to the bro community, but it is rather
> > popular.
> > >
> > > - The Bro team created the idea of bro packages, which can include bro
> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > > 'plugins' repo, they moved to have a 'packages' repo which is by
> default
> > > referenced by a bro-pkg tool they wrote for package management.
> > >
> > > - I believe I kicked this off (or at least I did in my head) when I
> > started
> > > complaining about the plugin divergence that occurred due to the move
> to
> > > bro/plugins (the right move at the time), but Metron's use of a local
> > > directory that hadn't been kept up to date. My current efforts are an
> > > attempt to make sure this doesn't happen again, and to take advantage
> of
> > > the bro-pkg benefits.
> > >
> > > - The gap between ~3/31 and actual progress on 11/12 is completely on
> me
> > -
> > > I had intentions of doing this work sooner but failed to do so.
> > >
> > > - You can most definitely still install/use the bro plugin without
> using
> > > bro-pkg. In fact, the README in my PR still has the instructions on how
> > to
> > > do so.
> > >
> > >
> > >
> > > Q1: The simple explanation is that the only thing that makes a plugin a
> > bro
> > > package is the inclusion of a bro-pkg.meta file, and it includes a
> > > build_command which could easily be manually performed to install by
> hand
> > > (assuming dependencies are met).
> > >
> > >
> > >
> > > I've worked with other projects that use submodules and while I'm fine
> > > discussing it, I suggest that we don't implement it. I put together a
> > quick
> > > example of why here[1], using the bro project as an example since it's
> > top
> > > of mind.
> > >
> > >
> > >
> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
> > > stopping a git clone && cd $dir && configure && make && make install,
> but
> > > using bro-pkg to install/load takes into account dependencies and unit
> > > tests when it is loaded (and thus fails early and more intuitively). It
> > > only must be a separate repo (or, more technically correct, a git
> branch
> > > that includes only the package) because of how bro-pkg works. If you'd
> > like
> > > to get an idea of how this would work in application for Bro users, you
> > can
> > > see my test instructions here (specifically step #3). If a 0.1 tag gets
> > > pushed to apache/metron-bro-plugin-kafka, the command could be
> `bro-pkg
> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force
> is
> > > just to remove user interaction, for an ansible spin-up).
> > >
> > >
> > >
> > >
> > >
> > > 1: To clone the Bro git repo, you must run `git clone --recursive
> > > https://github.com/bro/bro` <https://github.com/bro/bro> (note the
> > > --recursive). Not too big of a deal,
> > > but requires that you remember it and existing instructions/blog posts
> > may
> > > give users inaccurate steps. Let's make this worse and try to checkout
> > > their latest release, v2.5.2, and automatically update the submodules
> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
> fails
> > > because aux/plugins (https://github.com/bro/plugins) was removed since
> > > their latest release. Okay, we can work around this using `git checkout
> > > v2.5.2` and then remember to `git submodule update` every time you
> > checkout
> > > a release or branch. But because they have nested submodules, we
> actually
> > > need to run `git submodule update --recursive`. I can't imagine opting
> > into
> > > a workflow anything like this. There are other options as well, such as
> > git
> > > subtrees, but those I am less familiar with.
> > >
> > >
> > >
> > > Jon
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > >
> > > I am not sure that our use of the plugin necessarily equates to it
> being
> > > implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
> > > for an Apache project would be to make this available for use by the
> > > greater bro community.
> > > Unless we expect to do extensive iterative work on the plugin, which
> > would
> > > then make the decision to spin it out now premature.
> > >
> > > Then again, I might be wrong ;)
> > >
> > >
> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
> > >
> > > [Please pardon me that the below is a little labored. I’m trying to
> > > understand the implications for both release and use, which requires
> some
> > > explanation as well as the two questions needed. Q1 and Q2 below are
> > > probably the same question, asked in slightly different contexts.
> Please
> > > consider them together.]
> > >
> > > So this made me go back and look at the history that caused us to put
> the
> > > bro plugin in a separate repo. As best I can see, this was in
> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
> email
> > > discussion thread. Also please see
> > > https://issues.apache.org/jira/browse/METRON-883 for background on the
> > > plugin itself.
> > >
> > > As best I can assemble the many bits brought up in the threads, the
> > reasons
> > > to put it in a separate repo was:
> > > - The plugin was thought to be useful to multiple clients of bro and
> > kafka,
> > > including Storm and Spark, as well as Metron.
> > > - Originally the bro project was maintaining bro plugins and it was
> > thought
> > > they might adopt this one.
> > > - Bro then formalized their plugin framework BUT dumped all plugins out
> > of
> > > their sphere of maintenance.
> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
> > > that a package live within its own repo”. Jon said “the bro packages
> > model
> > > doesn't allow colocation with anything else.”
> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
> repo
> > > was created a few days later. But Metron wasn’t actually modified to
> > remove
> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
> > ago!
> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
> repo
> > > remain valid, if the bro plugin mechanism is used. But obviously there
> > are
> > > (non-conforming) ways to build the plugin as part of metron, and
> install
> > it
> > > in a way that works.
> > >
> > > Q1. I think that last statement needs some explanation. Nick or Jon,
> can
> > > you please expand on it, especially wrt how the end user installs the
> > > plugin once the plugin is built the two different ways? And whether
> it’s
> > > still valuable to have a separate repo for the plugin?
> > >
> > > Nick suggests using a submodule approach to managing the bro plugin,
> for
> > > Metron versioning purposes. As I understand it, this would continue the
> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
> > metron
> > > code tree for building, versioning, and release purposes. Git
> submodules
> > > are documented here: https://git-scm.com/book/en/
> v2/Git-Tools-Submodules
> > .
> > > We would use the submodule capability to clone the
> > metron-bro-plugin-kafka
> > > source code into a subdirectory of Metron at the time one clones the
> > metron
> > > repo. It would then be released with Metron as part of the source code
> > > release for a given version of Metron. Part of the way submodules are
> > > managed, is that git stores the SHA1 hash of the submodule into a file
> > > named .gitmodules, which in turn gets saved when you do a git push. So
> > > indeed submodules would ensure that everyone cloning a given version of
> > > metron would get the expected “version” (sha, actually) of
> > > metron-bro-plugin-kafka.
> > >
> > > This sounds like a good idea, although it isn’t without cost.
> Submodules
> > > impose the need for additional commands to actually get a copy of the
> > > submodule source, and if the plugin repo advanced beyond the version
> in a
> > > metron repo, it causes some ‘git status’ artifacts that could be
> > confusing
> > > to folks who aren’t familiar with submodules. But these can be
> > documented.
> > >
> > > Q2. Nick, what I’m not clear about is the process by which the
> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
> > > developers, and (b) end users. If it “must” be in a separate repo to be
> > > successfully built and managed by the bro plugin mechanism, does that
> > mean
> > > it can’t be built from the copy in the Metron source tree? Yet until
> > > November, that’s exactly what we were doing. Do we go back to doing
> that?
> > > What does that mean wrt users installing the plugin?
> > >
> > > Thanks for your patience in reading this far.
> > > --Matt
> > >
> > >
> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
> > >
> > > I agree with Nick. Since the plugin is tightly coupled with Metron why
> > not
> > > just pull it into the main repo and version it with the rest of the
> code?
> > > Do we really need the second repo for the plug-in?
> > >
> > > Thanks,
> > > James
> > >
> > >
> > >
> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
> > > >> I would suggest that we institute a release procedure for the
> package
> > > >> itself, but I don't think it necessarily has to line up with metron
> > > >> releases (happy to be persuaded otherwise). Then we can just link
> > metron
> > > >> to metron-bro-plugin-kafka by pointing to specific
> > > >> metron-bro-plugin-kafka releases (git tags
> > > >> <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > package.html#package-
> > > >> versioning>
> > > >> ).
> > > >> Right now, full-dev spins up against the
> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
> idea
> > > to
> > > >> have in place for an upcoming release. That is the crux of why I
> think
> > > we
> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
> before
> > > our
> > > >> next release (working on it as we speak).
> > > >> Jon
> > > >
> > > > ​I replayed Jon's comments from the other thread above.​
> > > >
> > > > My initial thought, is that I would not want to manage two separate
> > > release
> > > > processes. I don't want to have a roll call, cut release candidates
> and
> > > > test both.
> > > >
> > > > I was thinking we would just need to change some of the
> > behind-the-scenes
> > > > processes handled by the release manager. This is one area where I
> had
> > > > thought using a submodule in Git would help.
> > > >
> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> > > >
> > > >> + Restarting the thread to include mentors.
> > > >>
> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
> > external
> > > >> repository that we set up a while back.
> > > >>
> > > >> - Metron Core: git://git.apache.org/metron.git
> > > >> - Kafka Plugin for Bro: git://git.apache.org/
> > > >> metron-bro-plugin-kafka.git
> > > >>
> > > >> (Q) Do we need to change anything in the release procedure to
> account
> > > for
> > > >> this?
> > >
> > > -------------------
> > > Thank you,
> > >
> > > James Sirota
> > > PMC- Apache Metron
> > > jsirota AT apache DOT org
> > >
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Kyle Richardson <ky...@gmail.com>]

I agree with Justin. I suggest we move forward with option (a). I see it as
simpler and more flexible moving forward.

-Kyle

On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <ju...@gmail.com> wrote:

> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > >     - open a jira,
> > >
> > >     - add the submodule to the Metron code tree,
> > >
> > >     - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > >     - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen <ni...@nickallen.org>
> > > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" <de...@metron.apache.org>
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technical ones.
> > >
> > >
> > >
> > > (1) I do not want to change our release management process. While we
> > needed
> > > to make a new repo (a technical change), I did not want that to change
> > how
> > > we operate as a community (our procedures, policies, versioning and
> > release
> > > cycles).
> > >
> > >
> > >
> > > (2) I see no value in adopting a separate release management process
> for
> > > the Bro plugin alone. Having a separate release process does not make
> the
> > > plugin *more* available to the Bro community.
> > >
> > >
> > >
> > > (3) I also see no value in positioning the plugin to be spun-out of the
> > > Metron project. It is a part of Metron and I want to see it benefit and
> > > evolve "the Apache-way".
> > >
> > >
> > >
> > > In my mind, the best way to accommodate the additional repo, while
> > > minimizing changes to our release management process, is to treat the
> new
> > > repo as a submodule. I fail to see significant downsides to this
> > approach.
> > > A few extract command-line options do not seem overly onerous to me.
> > >
> > >
> > >
> > > Many thanks go to Jon for all the hard work he has put in enhancing the
> > > plugin.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > > In an attempt to keep this from becoming unbearably long, I will try to
> > > keep my responses short, but I would be happy to elaborate. That's a
> > fairly
> > > good timeline and summary, but here are some clarifications in
> > > corresponding order:
> > >
> > >
> > >
> > > - The plugin history is quite short and you can probably get a good bit
> > of
> > > context just by looking at the commits.
> > >
> > > - The plugin is only useful to the bro community, but it is rather
> > popular.
> > >
> > > - The Bro team created the idea of bro packages, which can include bro
> > > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > > 'plugins' repo, they moved to have a 'packages' repo which is by
> default
> > > referenced by a bro-pkg tool they wrote for package management.
> > >
> > > - I believe I kicked this off (or at least I did in my head) when I
> > started
> > > complaining about the plugin divergence that occurred due to the move
> to
> > > bro/plugins (the right move at the time), but Metron's use of a local
> > > directory that hadn't been kept up to date. My current efforts are an
> > > attempt to make sure this doesn't happen again, and to take advantage
> of
> > > the bro-pkg benefits.
> > >
> > > - The gap between ~3/31 and actual progress on 11/12 is completely on
> me
> > -
> > > I had intentions of doing this work sooner but failed to do so.
> > >
> > > - You can most definitely still install/use the bro plugin without
> using
> > > bro-pkg. In fact, the README in my PR still has the instructions on how
> > to
> > > do so.
> > >
> > >
> > >
> > > Q1: The simple explanation is that the only thing that makes a plugin a
> > bro
> > > package is the inclusion of a bro-pkg.meta file, and it includes a
> > > build_command which could easily be manually performed to install by
> hand
> > > (assuming dependencies are met).
> > >
> > >
> > >
> > > I've worked with other projects that use submodules and while I'm fine
> > > discussing it, I suggest that we don't implement it. I put together a
> > quick
> > > example of why here[1], using the bro project as an example since it's
> > top
> > > of mind.
> > >
> > >
> > >
> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
> > > stopping a git clone && cd $dir && configure && make && make install,
> but
> > > using bro-pkg to install/load takes into account dependencies and unit
> > > tests when it is loaded (and thus fails early and more intuitively). It
> > > only must be a separate repo (or, more technically correct, a git
> branch
> > > that includes only the package) because of how bro-pkg works. If you'd
> > like
> > > to get an idea of how this would work in application for Bro users, you
> > can
> > > see my test instructions here (specifically step #3). If a 0.1 tag gets
> > > pushed to apache/metron-bro-plugin-kafka, the command could be
> `bro-pkg
> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force
> is
> > > just to remove user interaction, for an ansible spin-up).
> > >
> > >
> > >
> > >
> > >
> > > 1: To clone the Bro git repo, you must run `git clone --recursive
> > > https://github.com/bro/bro` <https://github.com/bro/bro> (note the
> > > --recursive). Not too big of a deal,
> > > but requires that you remember it and existing instructions/blog posts
> > may
> > > give users inaccurate steps. Let's make this worse and try to checkout
> > > their latest release, v2.5.2, and automatically update the submodules
> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
> fails
> > > because aux/plugins (https://github.com/bro/plugins) was removed since
> > > their latest release. Okay, we can work around this using `git checkout
> > > v2.5.2` and then remember to `git submodule update` every time you
> > checkout
> > > a release or branch. But because they have nested submodules, we
> actually
> > > need to run `git submodule update --recursive`. I can't imagine opting
> > into
> > > a workflow anything like this. There are other options as well, such as
> > git
> > > subtrees, but those I am less familiar with.
> > >
> > >
> > >
> > > Jon
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
> > > wrote:
> > >
> > > I am not sure that our use of the plugin necessarily equates to it
> being
> > > implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
> > > for an Apache project would be to make this available for use by the
> > > greater bro community.
> > > Unless we expect to do extensive iterative work on the plugin, which
> > would
> > > then make the decision to spin it out now premature.
> > >
> > > Then again, I might be wrong ;)
> > >
> > >
> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
> > >
> > > [Please pardon me that the below is a little labored. I’m trying to
> > > understand the implications for both release and use, which requires
> some
> > > explanation as well as the two questions needed. Q1 and Q2 below are
> > > probably the same question, asked in slightly different contexts.
> Please
> > > consider them together.]
> > >
> > > So this made me go back and look at the history that caused us to put
> the
> > > bro plugin in a separate repo. As best I can see, this was in
> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
> email
> > > discussion thread. Also please see
> > > https://issues.apache.org/jira/browse/METRON-883 for background on the
> > > plugin itself.
> > >
> > > As best I can assemble the many bits brought up in the threads, the
> > reasons
> > > to put it in a separate repo was:
> > > - The plugin was thought to be useful to multiple clients of bro and
> > kafka,
> > > including Storm and Spark, as well as Metron.
> > > - Originally the bro project was maintaining bro plugins and it was
> > thought
> > > they might adopt this one.
> > > - Bro then formalized their plugin framework BUT dumped all plugins out
> > of
> > > their sphere of maintenance.
> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
> > > that a package live within its own repo”. Jon said “the bro packages
> > model
> > > doesn't allow colocation with anything else.”
> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
> repo
> > > was created a few days later. But Metron wasn’t actually modified to
> > remove
> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
> > ago!
> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
> repo
> > > remain valid, if the bro plugin mechanism is used. But obviously there
> > are
> > > (non-conforming) ways to build the plugin as part of metron, and
> install
> > it
> > > in a way that works.
> > >
> > > Q1. I think that last statement needs some explanation. Nick or Jon,
> can
> > > you please expand on it, especially wrt how the end user installs the
> > > plugin once the plugin is built the two different ways? And whether
> it’s
> > > still valuable to have a separate repo for the plugin?
> > >
> > > Nick suggests using a submodule approach to managing the bro plugin,
> for
> > > Metron versioning purposes. As I understand it, this would continue the
> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
> > metron
> > > code tree for building, versioning, and release purposes. Git
> submodules
> > > are documented here: https://git-scm.com/book/en/
> v2/Git-Tools-Submodules
> > .
> > > We would use the submodule capability to clone the
> > metron-bro-plugin-kafka
> > > source code into a subdirectory of Metron at the time one clones the
> > metron
> > > repo. It would then be released with Metron as part of the source code
> > > release for a given version of Metron. Part of the way submodules are
> > > managed, is that git stores the SHA1 hash of the submodule into a file
> > > named .gitmodules, which in turn gets saved when you do a git push. So
> > > indeed submodules would ensure that everyone cloning a given version of
> > > metron would get the expected “version” (sha, actually) of
> > > metron-bro-plugin-kafka.
> > >
> > > This sounds like a good idea, although it isn’t without cost.
> Submodules
> > > impose the need for additional commands to actually get a copy of the
> > > submodule source, and if the plugin repo advanced beyond the version
> in a
> > > metron repo, it causes some ‘git status’ artifacts that could be
> > confusing
> > > to folks who aren’t familiar with submodules. But these can be
> > documented.
> > >
> > > Q2. Nick, what I’m not clear about is the process by which the
> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
> > > developers, and (b) end users. If it “must” be in a separate repo to be
> > > successfully built and managed by the bro plugin mechanism, does that
> > mean
> > > it can’t be built from the copy in the Metron source tree? Yet until
> > > November, that’s exactly what we were doing. Do we go back to doing
> that?
> > > What does that mean wrt users installing the plugin?
> > >
> > > Thanks for your patience in reading this far.
> > > --Matt
> > >
> > >
> > > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
> > >
> > > I agree with Nick. Since the plugin is tightly coupled with Metron why
> > not
> > > just pull it into the main repo and version it with the rest of the
> code?
> > > Do we really need the second repo for the plug-in?
> > >
> > > Thanks,
> > > James
> > >
> > >
> > >
> > > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
> > > >> I would suggest that we institute a release procedure for the
> package
> > > >> itself, but I don't think it necessarily has to line up with metron
> > > >> releases (happy to be persuaded otherwise). Then we can just link
> > metron
> > > >> to metron-bro-plugin-kafka by pointing to specific
> > > >> metron-bro-plugin-kafka releases (git tags
> > > >> <
> > > http://bro-package-manager.readthedocs.io/en/stable/
> > package.html#package-
> > > >> versioning>
> > > >> ).
> > > >> Right now, full-dev spins up against the
> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good
> idea
> > > to
> > > >> have in place for an upcoming release. That is the crux of why I
> think
> > > we
> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
> before
> > > our
> > > >> next release (working on it as we speak).
> > > >> Jon
> > > >
> > > > ​I replayed Jon's comments from the other thread above.​
> > > >
> > > > My initial thought, is that I would not want to manage two separate
> > > release
> > > > processes. I don't want to have a roll call, cut release candidates
> and
> > > > test both.
> > > >
> > > > I was thinking we would just need to change some of the
> > behind-the-scenes
> > > > processes handled by the release manager. This is one area where I
> had
> > > > thought using a submodule in Git would help.
> > > >
> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> > > >
> > > >> + Restarting the thread to include mentors.
> > > >>
> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
> > external
> > > >> repository that we set up a while back.
> > > >>
> > > >> - Metron Core: git://git.apache.org/metron.git
> > > >> - Kafka Plugin for Bro: git://git.apache.org/
> > > >> metron-bro-plugin-kafka.git
> > > >>
> > > >> (Q) Do we need to change anything in the release procedure to
> account
> > > for
> > > >> this?
> > >
> > > -------------------
> > > Thank you,
> > >
> > > James Sirota
> > > PMC- Apache Metron
> > > jsirota AT apache DOT org
> > >
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Justin Leet <ju...@gmail.com>]

Do we have any further discussion on this?  Pardon me if I misstate
anyone's position, but it seems like we have a couple people (Otto and Jon
and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
section of people like myself without a particular horse in the race.

It seems like we need to come to some sort of consensus so that we can get
the release bus moving again, and right now it seems like (a) is gathering
more explicit support.  Do we have a compelling reason to not do (a)? To be
honest, my main worry is more "If we do (a) are we going to be miserable if
we need to iterate or adjust?" I'm not seeing anything that suggests
anything too terrible, so unless we see some more discussion, I suggest we
move forward with (a).

On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:

> I would prefer a, but I was initially thinking of doing the plugin first
> and then get in the two PRs out to use this new tag, which are already +1'd
> and just waiting on this conversation.  For reference,
> https://github.com/apache/metron/pull/847 and
> https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> Jon
>
> On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:
>
> > It seems to me, as I believe I have stated before that a) feels like the
> > proper way to handle this.  It is how I have seen other projects like
> NiFi
> > handle things as well.
> >
> >
> >
> > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
> >
> > Okay, looking at this from the perspective of making a release:
> >
> >
> >
> > We have two choices:
> >
> > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > metron-bro-plugin-kafka, at the same time and using the same process
> > (modulo the necessary) as Metron.  This is dirt simple.
> >
> > b) I or someone needs to:
> >
> >     - open a jira,
> >
> >     - add the submodule to the Metron code tree,
> >
> >     - possibly (optionally) add build mechanism to the maven poms, and
> >
> >     - document as much as we think appropriate regarding what it is, how
> to
> > build it, and how to update it,
> >
> > and commit that before the 0.4.2 release.
> >
> >
> >
> > What is the will of the community?
> >
> > Thanks,
> >
> > --Matt
> >
> >
> >
> > From: Nick Allen <ni...@nickallen.org>
> > Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
> > Date: Tuesday, November 28, 2017 at 9:09 AM
> > To: "dev@metron.apache.org" <de...@metron.apache.org>
> > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> Bro'
> >
> >
> >
> > I'll add a bit to Jon's technical comments.
> >
> >
> >
> > * We only created a separate repo because it was a technical requirement
> to
> > leverage the bro-pkg mechanism.
> >
> > * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> > Jon.
> >
> > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> the
> > plugin exactly how we use to.
> >
> >
> >
> > While I agree with Jon's technical comments, I disagree with the
> > non-technical ones.
> >
> >
> >
> > (1) I do not want to change our release management process. While we
> needed
> > to make a new repo (a technical change), I did not want that to change
> how
> > we operate as a community (our procedures, policies, versioning and
> release
> > cycles).
> >
> >
> >
> > (2) I see no value in adopting a separate release management process for
> > the Bro plugin alone. Having a separate release process does not make the
> > plugin *more* available to the Bro community.
> >
> >
> >
> > (3) I also see no value in positioning the plugin to be spun-out of the
> > Metron project. It is a part of Metron and I want to see it benefit and
> > evolve "the Apache-way".
> >
> >
> >
> > In my mind, the best way to accommodate the additional repo, while
> > minimizing changes to our release management process, is to treat the new
> > repo as a submodule. I fail to see significant downsides to this
> approach.
> > A few extract command-line options do not seem overly onerous to me.
> >
> >
> >
> > Many thanks go to Jon for all the hard work he has put in enhancing the
> > plugin.
> >
> >
> >
> >
> >
> > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> > In an attempt to keep this from becoming unbearably long, I will try to
> > keep my responses short, but I would be happy to elaborate. That's a
> fairly
> > good timeline and summary, but here are some clarifications in
> > corresponding order:
> >
> >
> >
> > - The plugin history is quite short and you can probably get a good bit
> of
> > context just by looking at the commits.
> >
> > - The plugin is only useful to the bro community, but it is rather
> popular.
> >
> > - The Bro team created the idea of bro packages, which can include bro
> > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > 'plugins' repo, they moved to have a 'packages' repo which is by default
> > referenced by a bro-pkg tool they wrote for package management.
> >
> > - I believe I kicked this off (or at least I did in my head) when I
> started
> > complaining about the plugin divergence that occurred due to the move to
> > bro/plugins (the right move at the time), but Metron's use of a local
> > directory that hadn't been kept up to date. My current efforts are an
> > attempt to make sure this doesn't happen again, and to take advantage of
> > the bro-pkg benefits.
> >
> > - The gap between ~3/31 and actual progress on 11/12 is completely on me
> -
> > I had intentions of doing this work sooner but failed to do so.
> >
> > - You can most definitely still install/use the bro plugin without using
> > bro-pkg. In fact, the README in my PR still has the instructions on how
> to
> > do so.
> >
> >
> >
> > Q1: The simple explanation is that the only thing that makes a plugin a
> bro
> > package is the inclusion of a bro-pkg.meta file, and it includes a
> > build_command which could easily be manually performed to install by hand
> > (assuming dependencies are met).
> >
> >
> >
> > I've worked with other projects that use submodules and while I'm fine
> > discussing it, I suggest that we don't implement it. I put together a
> quick
> > example of why here[1], using the bro project as an example since it's
> top
> > of mind.
> >
> >
> >
> > Q2: I think the answer to Q1 answers this. There is absolutely nothing
> > stopping a git clone && cd $dir && configure && make && make install, but
> > using bro-pkg to install/load takes into account dependencies and unit
> > tests when it is loaded (and thus fails early and more intuitively). It
> > only must be a separate repo (or, more technically correct, a git branch
> > that includes only the package) because of how bro-pkg works. If you'd
> like
> > to get an idea of how this would work in application for Bro users, you
> can
> > see my test instructions here (specifically step #3). If a 0.1 tag gets
> > pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
> > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
> > apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
> > just to remove user interaction, for an ansible spin-up).
> >
> >
> >
> >
> >
> > 1: To clone the Bro git repo, you must run `git clone --recursive
> > https://github.com/bro/bro` <https://github.com/bro/bro> (note the
> > --recursive). Not too big of a deal,
> > but requires that you remember it and existing instructions/blog posts
> may
> > give users inaccurate steps. Let's make this worse and try to checkout
> > their latest release, v2.5.2, and automatically update the submodules
> > appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
> > because aux/plugins (https://github.com/bro/plugins) was removed since
> > their latest release. Okay, we can work around this using `git checkout
> > v2.5.2` and then remember to `git submodule update` every time you
> checkout
> > a release or branch. But because they have nested submodules, we actually
> > need to run `git submodule update --recursive`. I can't imagine opting
> into
> > a workflow anything like this. There are other options as well, such as
> git
> > subtrees, but those I am less familiar with.
> >
> >
> >
> > Jon
> >
> >
> >
> > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
> > wrote:
> >
> > I am not sure that our use of the plugin necessarily equates to it being
> > implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
> > for an Apache project would be to make this available for use by the
> > greater bro community.
> > Unless we expect to do extensive iterative work on the plugin, which
> would
> > then make the decision to spin it out now premature.
> >
> > Then again, I might be wrong ;)
> >
> >
> > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
> >
> > [Please pardon me that the below is a little labored. I’m trying to
> > understand the implications for both release and use, which requires some
> > explanation as well as the two questions needed. Q1 and Q2 below are
> > probably the same question, asked in slightly different contexts. Please
> > consider them together.]
> >
> > So this made me go back and look at the history that caused us to put the
> > bro plugin in a separate repo. As best I can see, this was in
> > https://issues.apache.org/jira/browse/METRON-813 , which cites an email
> > discussion thread. Also please see
> > https://issues.apache.org/jira/browse/METRON-883 for background on the
> > plugin itself.
> >
> > As best I can assemble the many bits brought up in the threads, the
> reasons
> > to put it in a separate repo was:
> > - The plugin was thought to be useful to multiple clients of bro and
> kafka,
> > including Storm and Spark, as well as Metron.
> > - Originally the bro project was maintaining bro plugins and it was
> thought
> > they might adopt this one.
> > - Bro then formalized their plugin framework BUT dumped all plugins out
> of
> > their sphere of maintenance.
> > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
> > that a package live within its own repo”. Jon said “the bro packages
> model
> > doesn't allow colocation with anything else.”
> > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
> > was created a few days later. But Metron wasn’t actually modified to
> remove
> > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
> > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
> ago!
> > – with https://issues.apache.org/jira/browse/METRON-1309 .
> > - Presumably the need to have metron-bro-plugin-kafka in a separate repo
> > remain valid, if the bro plugin mechanism is used. But obviously there
> are
> > (non-conforming) ways to build the plugin as part of metron, and install
> it
> > in a way that works.
> >
> > Q1. I think that last statement needs some explanation. Nick or Jon, can
> > you please expand on it, especially wrt how the end user installs the
> > plugin once the plugin is built the two different ways? And whether it’s
> > still valuable to have a separate repo for the plugin?
> >
> > Nick suggests using a submodule approach to managing the bro plugin, for
> > Metron versioning purposes. As I understand it, this would continue the
> > existence of the metron-bro-plugin-kafka repo, but copy it into the
> metron
> > code tree for building, versioning, and release purposes. Git submodules
> > are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules
> .
> > We would use the submodule capability to clone the
> metron-bro-plugin-kafka
> > source code into a subdirectory of Metron at the time one clones the
> metron
> > repo. It would then be released with Metron as part of the source code
> > release for a given version of Metron. Part of the way submodules are
> > managed, is that git stores the SHA1 hash of the submodule into a file
> > named .gitmodules, which in turn gets saved when you do a git push. So
> > indeed submodules would ensure that everyone cloning a given version of
> > metron would get the expected “version” (sha, actually) of
> > metron-bro-plugin-kafka.
> >
> > This sounds like a good idea, although it isn’t without cost. Submodules
> > impose the need for additional commands to actually get a copy of the
> > submodule source, and if the plugin repo advanced beyond the version in a
> > metron repo, it causes some ‘git status’ artifacts that could be
> confusing
> > to folks who aren’t familiar with submodules. But these can be
> documented.
> >
> > Q2. Nick, what I’m not clear about is the process by which the
> > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
> > developers, and (b) end users. If it “must” be in a separate repo to be
> > successfully built and managed by the bro plugin mechanism, does that
> mean
> > it can’t be built from the copy in the Metron source tree? Yet until
> > November, that’s exactly what we were doing. Do we go back to doing that?
> > What does that mean wrt users installing the plugin?
> >
> > Thanks for your patience in reading this far.
> > --Matt
> >
> >
> > On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
> >
> > I agree with Nick. Since the plugin is tightly coupled with Metron why
> not
> > just pull it into the main repo and version it with the rest of the code?
> > Do we really need the second repo for the plug-in?
> >
> > Thanks,
> > James
> >
> >
> >
> > 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
> > >> I would suggest that we institute a release procedure for the package
> > >> itself, but I don't think it necessarily has to line up with metron
> > >> releases (happy to be persuaded otherwise). Then we can just link
> metron
> > >> to metron-bro-plugin-kafka by pointing to specific
> > >> metron-bro-plugin-kafka releases (git tags
> > >> <
> > http://bro-package-manager.readthedocs.io/en/stable/
> package.html#package-
> > >> versioning>
> > >> ).
> > >> Right now, full-dev spins up against the
> > >> apache/metron-bro-plugin-kafka master branch, which is not a good idea
> > to
> > >> have in place for an upcoming release. That is the crux of why I think
> > we
> > >> need to finalize the move to bro 2.5.2 and the plugin packaging before
> > our
> > >> next release (working on it as we speak).
> > >> Jon
> > >
> > > ​I replayed Jon's comments from the other thread above.​
> > >
> > > My initial thought, is that I would not want to manage two separate
> > release
> > > processes. I don't want to have a roll call, cut release candidates and
> > > test both.
> > >
> > > I was thinking we would just need to change some of the
> behind-the-scenes
> > > processes handled by the release manager. This is one area where I had
> > > thought using a submodule in Git would help.
> > >
> > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org>
> wrote:
> > >
> > >> + Restarting the thread to include mentors.
> > >>
> > >> The code of the 'Kafka Plugin for Bro' is now maintained in the
> external
> > >> repository that we set up a while back.
> > >>
> > >> - Metron Core: git://git.apache.org/metron.git
> > >> - Kafka Plugin for Bro: git://git.apache.org/
> > >> metron-bro-plugin-kafka.git
> > >>
> > >> (Q) Do we need to change anything in the release procedure to account
> > for
> > >> this?
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
> > --
> >
> > Jon
> >
> --
>
> Jon
>

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

["Zeolla@GMail.com" <ze...@gmail.com>]

I would prefer a, but I was initially thinking of doing the plugin first
and then get in the two PRs out to use this new tag, which are already +1'd
and just waiting on this conversation.  For reference,
https://github.com/apache/metron/pull/847 and
https://github.com/apache/metron-bro-plugin-kafka/pull/4

Jon

On Mon, Dec 4, 2017, 20:54 Otto Fowler <ot...@gmail.com> wrote:

> It seems to me, as I believe I have stated before that a) feels like the
> proper way to handle this.  It is how I have seen other projects like NiFi
> handle things as well.
>
>
>
> On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
>
> Okay, looking at this from the perspective of making a release:
>
>
>
> We have two choices:
>
> a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> metron-bro-plugin-kafka, at the same time and using the same process
> (modulo the necessary) as Metron.  This is dirt simple.
>
> b) I or someone needs to:
>
>     - open a jira,
>
>     - add the submodule to the Metron code tree,
>
>     - possibly (optionally) add build mechanism to the maven poms, and
>
>     - document as much as we think appropriate regarding what it is, how to
> build it, and how to update it,
>
> and commit that before the 0.4.2 release.
>
>
>
> What is the will of the community?
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen <ni...@nickallen.org>
> Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
> Date: Tuesday, November 28, 2017 at 9:09 AM
> To: "dev@metron.apache.org" <de...@metron.apache.org>
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I'll add a bit to Jon's technical comments.
>
>
>
> * We only created a separate repo because it was a technical requirement to
> leverage the bro-pkg mechanism.
>
> * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> Jon.
>
> * Enabling the bro-pkg mechanism is backwards compatible. I can install the
> plugin exactly how we use to.
>
>
>
> While I agree with Jon's technical comments, I disagree with the
> non-technical ones.
>
>
>
> (1) I do not want to change our release management process. While we needed
> to make a new repo (a technical change), I did not want that to change how
> we operate as a community (our procedures, policies, versioning and release
> cycles).
>
>
>
> (2) I see no value in adopting a separate release management process for
> the Bro plugin alone. Having a separate release process does not make the
> plugin *more* available to the Bro community.
>
>
>
> (3) I also see no value in positioning the plugin to be spun-out of the
> Metron project. It is a part of Metron and I want to see it benefit and
> evolve "the Apache-way".
>
>
>
> In my mind, the best way to accommodate the additional repo, while
> minimizing changes to our release management process, is to treat the new
> repo as a submodule. I fail to see significant downsides to this approach.
> A few extract command-line options do not seem overly onerous to me.
>
>
>
> Many thanks go to Jon for all the hard work he has put in enhancing the
> plugin.
>
>
>
>
>
> On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> In an attempt to keep this from becoming unbearably long, I will try to
> keep my responses short, but I would be happy to elaborate. That's a fairly
> good timeline and summary, but here are some clarifications in
> corresponding order:
>
>
>
> - The plugin history is quite short and you can probably get a good bit of
> context just by looking at the commits.
>
> - The plugin is only useful to the bro community, but it is rather popular.
>
> - The Bro team created the idea of bro packages, which can include bro
> plugins, bro scripts, or BroControl plugins. So, instead of having a
> 'plugins' repo, they moved to have a 'packages' repo which is by default
> referenced by a bro-pkg tool they wrote for package management.
>
> - I believe I kicked this off (or at least I did in my head) when I started
> complaining about the plugin divergence that occurred due to the move to
> bro/plugins (the right move at the time), but Metron's use of a local
> directory that hadn't been kept up to date. My current efforts are an
> attempt to make sure this doesn't happen again, and to take advantage of
> the bro-pkg benefits.
>
> - The gap between ~3/31 and actual progress on 11/12 is completely on me -
> I had intentions of doing this work sooner but failed to do so.
>
> - You can most definitely still install/use the bro plugin without using
> bro-pkg. In fact, the README in my PR still has the instructions on how to
> do so.
>
>
>
> Q1: The simple explanation is that the only thing that makes a plugin a bro
> package is the inclusion of a bro-pkg.meta file, and it includes a
> build_command which could easily be manually performed to install by hand
> (assuming dependencies are met).
>
>
>
> I've worked with other projects that use submodules and while I'm fine
> discussing it, I suggest that we don't implement it. I put together a quick
> example of why here[1], using the bro project as an example since it's top
> of mind.
>
>
>
> Q2: I think the answer to Q1 answers this. There is absolutely nothing
> stopping a git clone && cd $dir && configure && make && make install, but
> using bro-pkg to install/load takes into account dependencies and unit
> tests when it is loaded (and thus fails early and more intuitively). It
> only must be a separate repo (or, more technically correct, a git branch
> that includes only the package) because of how bro-pkg works. If you'd like
> to get an idea of how this would work in application for Bro users, you can
> see my test instructions here (specifically step #3). If a 0.1 tag gets
> pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
> install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
> apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
> just to remove user interaction, for an ansible spin-up).
>
>
>
>
>
> 1: To clone the Bro git repo, you must run `git clone --recursive
> https://github.com/bro/bro` <https://github.com/bro/bro> (note the
> --recursive). Not too big of a deal,
> but requires that you remember it and existing instructions/blog posts may
> give users inaccurate steps. Let's make this worse and try to checkout
> their latest release, v2.5.2, and automatically update the submodules
> appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
> because aux/plugins (https://github.com/bro/plugins) was removed since
> their latest release. Okay, we can work around this using `git checkout
> v2.5.2` and then remember to `git submodule update` every time you checkout
> a release or branch. But because they have nested submodules, we actually
> need to run `git submodule update --recursive`. I can't imagine opting into
> a workflow anything like this. There are other options as well, such as git
> subtrees, but those I am less familiar with.
>
>
>
> Jon
>
>
>
> On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
> wrote:
>
> I am not sure that our use of the plugin necessarily equates to it being
> implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
> for an Apache project would be to make this available for use by the
> greater bro community.
> Unless we expect to do extensive iterative work on the plugin, which would
> then make the decision to spin it out now premature.
>
> Then again, I might be wrong ;)
>
>
> On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
>
> [Please pardon me that the below is a little labored. I’m trying to
> understand the implications for both release and use, which requires some
> explanation as well as the two questions needed. Q1 and Q2 below are
> probably the same question, asked in slightly different contexts. Please
> consider them together.]
>
> So this made me go back and look at the history that caused us to put the
> bro plugin in a separate repo. As best I can see, this was in
> https://issues.apache.org/jira/browse/METRON-813 , which cites an email
> discussion thread. Also please see
> https://issues.apache.org/jira/browse/METRON-883 for background on the
> plugin itself.
>
> As best I can assemble the many bits brought up in the threads, the reasons
> to put it in a separate repo was:
> - The plugin was thought to be useful to multiple clients of bro and kafka,
> including Storm and Spark, as well as Metron.
> - Originally the bro project was maintaining bro plugins and it was thought
> they might adopt this one.
> - Bro then formalized their plugin framework BUT dumped all plugins out of
> their sphere of maintenance.
> - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
> that a package live within its own repo”. Jon said “the bro packages model
> doesn't allow colocation with anything else.”
> - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
> was created a few days later. But Metron wasn’t actually modified to remove
> the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
> plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
> – with https://issues.apache.org/jira/browse/METRON-1309 .
> - Presumably the need to have metron-bro-plugin-kafka in a separate repo
> remain valid, if the bro plugin mechanism is used. But obviously there are
> (non-conforming) ways to build the plugin as part of metron, and install it
> in a way that works.
>
> Q1. I think that last statement needs some explanation. Nick or Jon, can
> you please expand on it, especially wrt how the end user installs the
> plugin once the plugin is built the two different ways? And whether it’s
> still valuable to have a separate repo for the plugin?
>
> Nick suggests using a submodule approach to managing the bro plugin, for
> Metron versioning purposes. As I understand it, this would continue the
> existence of the metron-bro-plugin-kafka repo, but copy it into the metron
> code tree for building, versioning, and release purposes. Git submodules
> are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
> We would use the submodule capability to clone the metron-bro-plugin-kafka
> source code into a subdirectory of Metron at the time one clones the metron
> repo. It would then be released with Metron as part of the source code
> release for a given version of Metron. Part of the way submodules are
> managed, is that git stores the SHA1 hash of the submodule into a file
> named .gitmodules, which in turn gets saved when you do a git push. So
> indeed submodules would ensure that everyone cloning a given version of
> metron would get the expected “version” (sha, actually) of
> metron-bro-plugin-kafka.
>
> This sounds like a good idea, although it isn’t without cost. Submodules
> impose the need for additional commands to actually get a copy of the
> submodule source, and if the plugin repo advanced beyond the version in a
> metron repo, it causes some ‘git status’ artifacts that could be confusing
> to folks who aren’t familiar with submodules. But these can be documented.
>
> Q2. Nick, what I’m not clear about is the process by which the
> metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
> developers, and (b) end users. If it “must” be in a separate repo to be
> successfully built and managed by the bro plugin mechanism, does that mean
> it can’t be built from the copy in the Metron source tree? Yet until
> November, that’s exactly what we were doing. Do we go back to doing that?
> What does that mean wrt users installing the plugin?
>
> Thanks for your patience in reading this far.
> --Matt
>
>
> On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
>
> I agree with Nick. Since the plugin is tightly coupled with Metron why not
> just pull it into the main repo and version it with the rest of the code?
> Do we really need the second repo for the plug-in?
>
> Thanks,
> James
>
>
>
> 16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
> >> I would suggest that we institute a release procedure for the package
> >> itself, but I don't think it necessarily has to line up with metron
> >> releases (happy to be persuaded otherwise). Then we can just link metron
> >> to metron-bro-plugin-kafka by pointing to specific
> >> metron-bro-plugin-kafka releases (git tags
> >> <
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> >> versioning>
> >> ).
> >> Right now, full-dev spins up against the
> >> apache/metron-bro-plugin-kafka master branch, which is not a good idea
> to
> >> have in place for an upcoming release. That is the crux of why I think
> we
> >> need to finalize the move to bro 2.5.2 and the plugin packaging before
> our
> >> next release (working on it as we speak).
> >> Jon
> >
> > ​I replayed Jon's comments from the other thread above.​
> >
> > My initial thought, is that I would not want to manage two separate
> release
> > processes. I don't want to have a roll call, cut release candidates and
> > test both.
> >
> > I was thinking we would just need to change some of the behind-the-scenes
> > processes handled by the release manager. This is one area where I had
> > thought using a submodule in Git would help.
> >
> > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org> wrote:
> >
> >> + Restarting the thread to include mentors.
> >>
> >> The code of the 'Kafka Plugin for Bro' is now maintained in the external
> >> repository that we set up a while back.
> >>
> >> - Metron Core: git://git.apache.org/metron.git
> >> - Kafka Plugin for Bro: git://git.apache.org/
> >> metron-bro-plugin-kafka.git
> >>
> >> (Q) Do we need to change anything in the release procedure to account
> for
> >> this?
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
> --
>
> Jon
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley <ma...@apache.org>]

Honestly, I think (a) is sufficient, and less work.  But I’m willing to do (b) if the community prefers.


On 12/4/17, 5:54 PM, "Otto Fowler" <ot...@gmail.com> wrote:

    It seems to me, as I believe I have stated before that a) feels like the
    proper way to handle this.  It is how I have seen other projects like NiFi
    handle things as well.
    
    
    
    On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:
    
    Okay, looking at this from the perspective of making a release:
    
    
    
    We have two choices:
    
    a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
    metron-bro-plugin-kafka, at the same time and using the same process
    (modulo the necessary) as Metron.  This is dirt simple.
    
    b) I or someone needs to:
    
        - open a jira,
    
        - add the submodule to the Metron code tree,
    
        - possibly (optionally) add build mechanism to the maven poms, and
    
        - document as much as we think appropriate regarding what it is, how to
    build it, and how to update it,
    
    and commit that before the 0.4.2 release.
    
    
    
    What is the will of the community?
    
    Thanks,
    
    --Matt
    
    
    
    From: Nick Allen <ni...@nickallen.org>
    Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
    Date: Tuesday, November 28, 2017 at 9:09 AM
    To: "dev@metron.apache.org" <de...@metron.apache.org>
    Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
    
    
    
    I'll add a bit to Jon's technical comments.
    
    
    
    * We only created a separate repo because it was a technical requirement to
    leverage the bro-pkg mechanism.
    
    * Leveraging the new bro-pkg mechanism has many advantages as outlined by
    Jon.
    
    * Enabling the bro-pkg mechanism is backwards compatible. I can install the
    plugin exactly how we use to.
    
    
    
    While I agree with Jon's technical comments, I disagree with the
    non-technical ones.
    
    
    
    (1) I do not want to change our release management process. While we needed
    to make a new repo (a technical change), I did not want that to change how
    we operate as a community (our procedures, policies, versioning and release
    cycles).
    
    
    
    (2) I see no value in adopting a separate release management process for
    the Bro plugin alone. Having a separate release process does not make the
    plugin *more* available to the Bro community.
    
    
    
    (3) I also see no value in positioning the plugin to be spun-out of the
    Metron project. It is a part of Metron and I want to see it benefit and
    evolve "the Apache-way".
    
    
    
    In my mind, the best way to accommodate the additional repo, while
    minimizing changes to our release management process, is to treat the new
    repo as a submodule. I fail to see significant downsides to this approach.
    A few extract command-line options do not seem overly onerous to me.
    
    
    
    Many thanks go to Jon for all the hard work he has put in enhancing the
    plugin.
    
    
    
    
    
    On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
    wrote:
    
    In an attempt to keep this from becoming unbearably long, I will try to
    keep my responses short, but I would be happy to elaborate. That's a fairly
    good timeline and summary, but here are some clarifications in
    corresponding order:
    
    
    
    - The plugin history is quite short and you can probably get a good bit of
    context just by looking at the commits.
    
    - The plugin is only useful to the bro community, but it is rather popular.
    
    - The Bro team created the idea of bro packages, which can include bro
    plugins, bro scripts, or BroControl plugins. So, instead of having a
    'plugins' repo, they moved to have a 'packages' repo which is by default
    referenced by a bro-pkg tool they wrote for package management.
    
    - I believe I kicked this off (or at least I did in my head) when I started
    complaining about the plugin divergence that occurred due to the move to
    bro/plugins (the right move at the time), but Metron's use of a local
    directory that hadn't been kept up to date. My current efforts are an
    attempt to make sure this doesn't happen again, and to take advantage of
    the bro-pkg benefits.
    
    - The gap between ~3/31 and actual progress on 11/12 is completely on me -
    I had intentions of doing this work sooner but failed to do so.
    
    - You can most definitely still install/use the bro plugin without using
    bro-pkg. In fact, the README in my PR still has the instructions on how to
    do so.
    
    
    
    Q1: The simple explanation is that the only thing that makes a plugin a bro
    package is the inclusion of a bro-pkg.meta file, and it includes a
    build_command which could easily be manually performed to install by hand
    (assuming dependencies are met).
    
    
    
    I've worked with other projects that use submodules and while I'm fine
    discussing it, I suggest that we don't implement it. I put together a quick
    example of why here[1], using the bro project as an example since it's top
    of mind.
    
    
    
    Q2: I think the answer to Q1 answers this. There is absolutely nothing
    stopping a git clone && cd $dir && configure && make && make install, but
    using bro-pkg to install/load takes into account dependencies and unit
    tests when it is loaded (and thus fails early and more intuitively). It
    only must be a separate repo (or, more technically correct, a git branch
    that includes only the package) because of how bro-pkg works. If you'd like
    to get an idea of how this would work in application for Bro users, you can
    see my test instructions here (specifically step #3). If a 0.1 tag gets
    pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
    install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
    apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
    just to remove user interaction, for an ansible spin-up).
    
    
    
    
    
    1: To clone the Bro git repo, you must run `git clone --recursive
    https://github.com/bro/bro` (note the --recursive). Not too big of a deal,
    but requires that you remember it and existing instructions/blog posts may
    give users inaccurate steps. Let's make this worse and try to checkout
    their latest release, v2.5.2, and automatically update the submodules
    appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
    because aux/plugins (https://github.com/bro/plugins) was removed since
    their latest release. Okay, we can work around this using `git checkout
    v2.5.2` and then remember to `git submodule update` every time you checkout
    a release or branch. But because they have nested submodules, we actually
    need to run `git submodule update --recursive`. I can't imagine opting into
    a workflow anything like this. There are other options as well, such as git
    subtrees, but those I am less familiar with.
    
    
    
    Jon
    
    
    
    On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
    wrote:
    
    I am not sure that our use of the plugin necessarily equates to it being
    implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
    for an Apache project would be to make this available for use by the
    greater bro community.
    Unless we expect to do extensive iterative work on the plugin, which would
    then make the decision to spin it out now premature.
    
    Then again, I might be wrong ;)
    
    
    On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:
    
    [Please pardon me that the below is a little labored. I’m trying to
    understand the implications for both release and use, which requires some
    explanation as well as the two questions needed. Q1 and Q2 below are
    probably the same question, asked in slightly different contexts. Please
    consider them together.]
    
    So this made me go back and look at the history that caused us to put the
    bro plugin in a separate repo. As best I can see, this was in
    https://issues.apache.org/jira/browse/METRON-813 , which cites an email
    discussion thread. Also please see
    https://issues.apache.org/jira/browse/METRON-883 for background on the
    plugin itself.
    
    As best I can assemble the many bits brought up in the threads, the reasons
    to put it in a separate repo was:
    - The plugin was thought to be useful to multiple clients of bro and kafka,
    including Storm and Spark, as well as Metron.
    - Originally the bro project was maintaining bro plugins and it was thought
    they might adopt this one.
    - Bro then formalized their plugin framework BUT dumped all plugins out of
    their sphere of maintenance.
    - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
    that a package live within its own repo”. Jon said “the bro packages model
    doesn't allow colocation with anything else.”
    - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
    was created a few days later. But Metron wasn’t actually modified to remove
    the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
    plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
    – with https://issues.apache.org/jira/browse/METRON-1309 .
    - Presumably the need to have metron-bro-plugin-kafka in a separate repo
    remain valid, if the bro plugin mechanism is used. But obviously there are
    (non-conforming) ways to build the plugin as part of metron, and install it
    in a way that works.
    
    Q1. I think that last statement needs some explanation. Nick or Jon, can
    you please expand on it, especially wrt how the end user installs the
    plugin once the plugin is built the two different ways? And whether it’s
    still valuable to have a separate repo for the plugin?
    
    Nick suggests using a submodule approach to managing the bro plugin, for
    Metron versioning purposes. As I understand it, this would continue the
    existence of the metron-bro-plugin-kafka repo, but copy it into the metron
    code tree for building, versioning, and release purposes. Git submodules
    are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
    We would use the submodule capability to clone the metron-bro-plugin-kafka
    source code into a subdirectory of Metron at the time one clones the metron
    repo. It would then be released with Metron as part of the source code
    release for a given version of Metron. Part of the way submodules are
    managed, is that git stores the SHA1 hash of the submodule into a file
    named .gitmodules, which in turn gets saved when you do a git push. So
    indeed submodules would ensure that everyone cloning a given version of
    metron would get the expected “version” (sha, actually) of
    metron-bro-plugin-kafka.
    
    This sounds like a good idea, although it isn’t without cost. Submodules
    impose the need for additional commands to actually get a copy of the
    submodule source, and if the plugin repo advanced beyond the version in a
    metron repo, it causes some ‘git status’ artifacts that could be confusing
    to folks who aren’t familiar with submodules. But these can be documented.
    
    Q2. Nick, what I’m not clear about is the process by which the
    metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
    developers, and (b) end users. If it “must” be in a separate repo to be
    successfully built and managed by the bro plugin mechanism, does that mean
    it can’t be built from the copy in the Metron source tree? Yet until
    November, that’s exactly what we were doing. Do we go back to doing that?
    What does that mean wrt users installing the plugin?
    
    Thanks for your patience in reading this far.
    --Matt
    
    
    On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:
    
    I agree with Nick. Since the plugin is tightly coupled with Metron why not
    just pull it into the main repo and version it with the rest of the code?
    Do we really need the second repo for the plug-in?
    
    Thanks,
    James
    
    
    
    16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
    >> I would suggest that we institute a release procedure for the package
    >> itself, but I don't think it necessarily has to line up with metron
    >> releases (happy to be persuaded otherwise). Then we can just link metron
    >> to metron-bro-plugin-kafka by pointing to specific
    >> metron-bro-plugin-kafka releases (git tags
    >> <
    http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
    >> versioning>
    >> ).
    >> Right now, full-dev spins up against the
    >> apache/metron-bro-plugin-kafka master branch, which is not a good idea
    to
    >> have in place for an upcoming release. That is the crux of why I think
    we
    >> need to finalize the move to bro 2.5.2 and the plugin packaging before
    our
    >> next release (working on it as we speak).
    >> Jon
    >
    > ​I replayed Jon's comments from the other thread above.​
    >
    > My initial thought, is that I would not want to manage two separate
    release
    > processes. I don't want to have a roll call, cut release candidates and
    > test both.
    >
    > I was thinking we would just need to change some of the behind-the-scenes
    > processes handled by the release manager. This is one area where I had
    > thought using a submodule in Git would help.
    >
    > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org> wrote:
    >
    >> + Restarting the thread to include mentors.
    >>
    >> The code of the 'Kafka Plugin for Bro' is now maintained in the external
    >> repository that we set up a while back.
    >>
    >> - Metron Core: git://git.apache.org/metron.git
    >> - Kafka Plugin for Bro: git://git.apache.org/
    >> metron-bro-plugin-kafka.git
    >>
    >> (Q) Do we need to change anything in the release procedure to account
    for
    >> this?
    
    -------------------
    Thank you,
    
    James Sirota
    PMC- Apache Metron
    jsirota AT apache DOT org
    
    -- 
    
    Jon
    

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Otto Fowler <ot...@gmail.com>]

It seems to me, as I believe I have stated before that a) feels like the
proper way to handle this.  It is how I have seen other projects like NiFi
handle things as well.



On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org) wrote:

Okay, looking at this from the perspective of making a release:



We have two choices:

a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
metron-bro-plugin-kafka, at the same time and using the same process
(modulo the necessary) as Metron.  This is dirt simple.

b) I or someone needs to:

    - open a jira,

    - add the submodule to the Metron code tree,

    - possibly (optionally) add build mechanism to the maven poms, and

    - document as much as we think appropriate regarding what it is, how to
build it, and how to update it,

and commit that before the 0.4.2 release.



What is the will of the community?

Thanks,

--Matt



From: Nick Allen <ni...@nickallen.org>
Reply-To: "dev@metron.apache.org" <de...@metron.apache.org>
Date: Tuesday, November 28, 2017 at 9:09 AM
To: "dev@metron.apache.org" <de...@metron.apache.org>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'



I'll add a bit to Jon's technical comments.



* We only created a separate repo because it was a technical requirement to
leverage the bro-pkg mechanism.

* Leveraging the new bro-pkg mechanism has many advantages as outlined by
Jon.

* Enabling the bro-pkg mechanism is backwards compatible. I can install the
plugin exactly how we use to.



While I agree with Jon's technical comments, I disagree with the
non-technical ones.



(1) I do not want to change our release management process. While we needed
to make a new repo (a technical change), I did not want that to change how
we operate as a community (our procedures, policies, versioning and release
cycles).



(2) I see no value in adopting a separate release management process for
the Bro plugin alone. Having a separate release process does not make the
plugin *more* available to the Bro community.



(3) I also see no value in positioning the plugin to be spun-out of the
Metron project. It is a part of Metron and I want to see it benefit and
evolve "the Apache-way".



In my mind, the best way to accommodate the additional repo, while
minimizing changes to our release management process, is to treat the new
repo as a submodule. I fail to see significant downsides to this approach.
A few extract command-line options do not seem overly onerous to me.



Many thanks go to Jon for all the hard work he has put in enhancing the
plugin.





On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <ze...@gmail.com>
wrote:

In an attempt to keep this from becoming unbearably long, I will try to
keep my responses short, but I would be happy to elaborate. That's a fairly
good timeline and summary, but here are some clarifications in
corresponding order:



- The plugin history is quite short and you can probably get a good bit of
context just by looking at the commits.

- The plugin is only useful to the bro community, but it is rather popular.

- The Bro team created the idea of bro packages, which can include bro
plugins, bro scripts, or BroControl plugins. So, instead of having a
'plugins' repo, they moved to have a 'packages' repo which is by default
referenced by a bro-pkg tool they wrote for package management.

- I believe I kicked this off (or at least I did in my head) when I started
complaining about the plugin divergence that occurred due to the move to
bro/plugins (the right move at the time), but Metron's use of a local
directory that hadn't been kept up to date. My current efforts are an
attempt to make sure this doesn't happen again, and to take advantage of
the bro-pkg benefits.

- The gap between ~3/31 and actual progress on 11/12 is completely on me -
I had intentions of doing this work sooner but failed to do so.

- You can most definitely still install/use the bro plugin without using
bro-pkg. In fact, the README in my PR still has the instructions on how to
do so.



Q1: The simple explanation is that the only thing that makes a plugin a bro
package is the inclusion of a bro-pkg.meta file, and it includes a
build_command which could easily be manually performed to install by hand
(assuming dependencies are met).



I've worked with other projects that use submodules and while I'm fine
discussing it, I suggest that we don't implement it. I put together a quick
example of why here[1], using the bro project as an example since it's top
of mind.



Q2: I think the answer to Q1 answers this. There is absolutely nothing
stopping a git clone && cd $dir && configure && make && make install, but
using bro-pkg to install/load takes into account dependencies and unit
tests when it is loaded (and thus fails early and more intuitively). It
only must be a separate repo (or, more technically correct, a git branch
that includes only the package) because of how bro-pkg works. If you'd like
to get an idea of how this would work in application for Bro users, you can
see my test instructions here (specifically step #3). If a 0.1 tag gets
pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
just to remove user interaction, for an ansible spin-up).





1: To clone the Bro git repo, you must run `git clone --recursive
https://github.com/bro/bro` (note the --recursive). Not too big of a deal,
but requires that you remember it and existing instructions/blog posts may
give users inaccurate steps. Let's make this worse and try to checkout
their latest release, v2.5.2, and automatically update the submodules
appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
because aux/plugins (https://github.com/bro/plugins) was removed since
their latest release. Okay, we can work around this using `git checkout
v2.5.2` and then remember to `git submodule update` every time you checkout
a release or branch. But because they have nested submodules, we actually
need to run `git submodule update --recursive`. I can't imagine opting into
a workflow anything like this. There are other options as well, such as git
subtrees, but those I am less familiar with.



Jon



On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ot...@gmail.com>
wrote:

I am not sure that our use of the plugin necessarily equates to it being
implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
for an Apache project would be to make this available for use by the
greater bro community.
Unless we expect to do extensive iterative work on the plugin, which would
then make the decision to spin it out now premature.

Then again, I might be wrong ;)


On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:

[Please pardon me that the below is a little labored. I’m trying to
understand the implications for both release and use, which requires some
explanation as well as the two questions needed. Q1 and Q2 below are
probably the same question, asked in slightly different contexts. Please
consider them together.]

So this made me go back and look at the history that caused us to put the
bro plugin in a separate repo. As best I can see, this was in
https://issues.apache.org/jira/browse/METRON-813 , which cites an email
discussion thread. Also please see
https://issues.apache.org/jira/browse/METRON-883 for background on the
plugin itself.

As best I can assemble the many bits brought up in the threads, the reasons
to put it in a separate repo was:
- The plugin was thought to be useful to multiple clients of bro and kafka,
including Storm and Spark, as well as Metron.
- Originally the bro project was maintaining bro plugins and it was thought
they might adopt this one.
- Bro then formalized their plugin framework BUT dumped all plugins out of
their sphere of maintenance.
- As of 3/31/2017, Nick said that “the [bro] package mechanism requires
that a package live within its own repo”. Jon said “the bro packages model
doesn't allow colocation with anything else.”
- So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
was created a few days later. But Metron wasn’t actually modified to remove
the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
– with https://issues.apache.org/jira/browse/METRON-1309 .
- Presumably the need to have metron-bro-plugin-kafka in a separate repo
remain valid, if the bro plugin mechanism is used. But obviously there are
(non-conforming) ways to build the plugin as part of metron, and install it
in a way that works.

Q1. I think that last statement needs some explanation. Nick or Jon, can
you please expand on it, especially wrt how the end user installs the
plugin once the plugin is built the two different ways? And whether it’s
still valuable to have a separate repo for the plugin?

Nick suggests using a submodule approach to managing the bro plugin, for
Metron versioning purposes. As I understand it, this would continue the
existence of the metron-bro-plugin-kafka repo, but copy it into the metron
code tree for building, versioning, and release purposes. Git submodules
are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
We would use the submodule capability to clone the metron-bro-plugin-kafka
source code into a subdirectory of Metron at the time one clones the metron
repo. It would then be released with Metron as part of the source code
release for a given version of Metron. Part of the way submodules are
managed, is that git stores the SHA1 hash of the submodule into a file
named .gitmodules, which in turn gets saved when you do a git push. So
indeed submodules would ensure that everyone cloning a given version of
metron would get the expected “version” (sha, actually) of
metron-bro-plugin-kafka.

This sounds like a good idea, although it isn’t without cost. Submodules
impose the need for additional commands to actually get a copy of the
submodule source, and if the plugin repo advanced beyond the version in a
metron repo, it causes some ‘git status’ artifacts that could be confusing
to folks who aren’t familiar with submodules. But these can be documented.

Q2. Nick, what I’m not clear about is the process by which the
metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
developers, and (b) end users. If it “must” be in a separate repo to be
successfully built and managed by the bro plugin mechanism, does that mean
it can’t be built from the copy in the Metron source tree? Yet until
November, that’s exactly what we were doing. Do we go back to doing that?
What does that mean wrt users installing the plugin?

Thanks for your patience in reading this far.
--Matt


On 11/27/17, 2:58 PM, "James Sirota" <js...@apache.org> wrote:

I agree with Nick. Since the plugin is tightly coupled with Metron why not
just pull it into the main repo and version it with the rest of the code?
Do we really need the second repo for the plug-in?

Thanks,
James



16.11.2017, 08:06, "Nick Allen" <ni...@nickallen.org>:
>> I would suggest that we institute a release procedure for the package
>> itself, but I don't think it necessarily has to line up with metron
>> releases (happy to be persuaded otherwise). Then we can just link metron
>> to metron-bro-plugin-kafka by pointing to specific
>> metron-bro-plugin-kafka releases (git tags
>> <
http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
>> versioning>
>> ).
>> Right now, full-dev spins up against the
>> apache/metron-bro-plugin-kafka master branch, which is not a good idea
to
>> have in place for an upcoming release. That is the crux of why I think
we
>> need to finalize the move to bro 2.5.2 and the plugin packaging before
our
>> next release (working on it as we speak).
>> Jon
>
> ​I replayed Jon's comments from the other thread above.​
>
> My initial thought, is that I would not want to manage two separate
release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> + Restarting the thread to include mentors.
>>
>> The code of the 'Kafka Plugin for Bro' is now maintained in the external
>> repository that we set up a while back.
>>
>> - Metron Core: git://git.apache.org/metron.git
>> - Kafka Plugin for Bro: git://git.apache.org/
>> metron-bro-plugin-kafka.git
>>
>> (Q) Do we need to change anything in the release procedure to account
for
>> this?

-------------------
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

-- 

Jon