resolution on the build / project source structure

[Joe Witt <jo...@gmail.com>]

Folks,

We need to come to a conclusion on what we'll do about the build process
specifically as it relates to the source structure - if anything.  The
longer we wait the more disruptive and confusing it will be especially for
potential new contributors.

Background / Components / Points of tension:
- The 'Nifi API': This is a very small library which defines the interfaces
and some very basic abstract implementations or highly common functions.
The API is critical to building extensions and critical to the function of
the core application.
Frequency of change: Should be rare

- The core application: This is the implementation of the data flow engine,
the web services, class loading,
Frequency of change: Routine.

- Utilities: Things used by both the core and the extensions
Frequency of change: Routine

- Extensions: Processors, Prioritizers, Custom UIs, Reporting Tasks,
Controller tasks, etc... . These are the things we want folks to be able to
build - the intentional points of extension.
Frequency of change: Constant

As I think through these different concerns and how maven works it isn't
clear to me what a better model would be.  Having not yet gone through the
challenges of an official apache release I don''t appreciate yet what pain
points that brings either.

Our current model requires someone to know a lot of ordering for the
build.  But if we put a CI server behind it that would be less of an issue
except for those few folks that want to build the whole thing by hand.  For
that perhaps the build-order is sufficient.

So - we leaving it as is or is there a good proposal?

Thanks
Joe

Re: resolution on the build / project source structure

[Benson Margulies <bi...@gmail.com>]

There are working patterns in my experience.

Pattern #1:

Use maven-release-plugin-2.5.1 that allows multiple releasable items in a
single repo.

Then you can have a shared parent pom in its own directory (with no
modules) that you release, and then a series of independently released
components sitting next to it, and each has the appropriate release plugin
configuration.

Pattern #2:

One glorious gigantic monolith -- a parent with modules -- released all at
once.

Pattern #3:

Multiple git repos, one for each releasable item.

Re: resolution on the build / project source structure

[Joe Witt <jo...@gmail.com>]

Joey, Sean,

Frequency of change was just a hint/nod to how often we'd want to release
those components in particular.  I know there is the option of a single
parent pom (which we have) with modules (which is what I think you were
getting at).  But what I had found in past practice was that maven handling
of that as complexity rose was not very good particularly when wanting to
use the release plugin.  Now, a lot of time has passed and as Sean notes
perhaps now this is more feasible.  Perhaps I'll take a stab at this off to
the side and see what I find.  We definitely don't want to have to bump all
versions all the time given that some parts of the application change (or
we'd want them to) quite often.  I do agree commons is a potentially good
model to look after.  I was toying with this concept a bit early on in the
incubator proposal where we'd have sub-projects.

Thanks
Joe

On Fri, Dec 12, 2014 at 1:36 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> Not to get into minutiae too much, but you can have a common parent pom and
> still version / release independently. As an example, many (most?) ASF
> projects use the Apache root pom as their parent, and that changes
> infrequently. That said, I think the build mechanics part of the issue
> isn't as important as the varying release cycles. We won't be the first ASF
> project to use different builds for different components and a unifying
> build script or makefile. (though I don't recall seeing one where those
> different component builds are all actually maven)
>
> I think the big cost for keeping the releases independent will be the
> overhead of running all those release VOTEs, since you need to have at
> least 3 PMCs sign off on release correctness (independent of the
> correctness of the software in the release). It might be worth looking at
> how things work over in Apache Commons, since they have a bunch of
> differently versioned components.
>
> The most important bit is that incubator is the time to figure this sort of
> thing out. There's little risk in trying to maintain the current habit of
> different release cycles. If it gets to be too much of a burden, we can
> just adapt later. We could always adopt a versioning scheme that provides
> different levels of compatibility guarantees for different kinds of
> components. For example, in Avro's X.Y.Z versioning, X only increments if
> data stored to disk can't be backwards compatibly read and Y increments for
> breaking changes in the language specific implementation APIs.
>
>
>
> On Fri, Dec 12, 2014 at 12:10 PM, Joey Echeverria <jo...@cloudera.com>
> wrote:
>
> > When you say "Frequency of change", does that just refer to the level
> > of development activity or do you expect different components to have
> > different release cycles?
> >
> > I'm assuming the reason there isn't currently a parent POM with each
> > of those just maven modules under the parent is because of the desire
> > to version/release them independently.
> >
> > -Joey
> >
> > On Fri, Dec 12, 2014 at 6:37 AM, Joe Witt <jo...@gmail.com> wrote:
> > > Folks,
> > >
> > > We need to come to a conclusion on what we'll do about the build
> process
> > > specifically as it relates to the source structure - if anything.  The
> > > longer we wait the more disruptive and confusing it will be especially
> > for
> > > potential new contributors.
> > >
> > > Background / Components / Points of tension:
> > > - The 'Nifi API': This is a very small library which defines the
> > interfaces
> > > and some very basic abstract implementations or highly common
> functions.
> > > The API is critical to building extensions and critical to the function
> > of
> > > the core application.
> > > Frequency of change: Should be rare
> > >
> > > - The core application: This is the implementation of the data flow
> > engine,
> > > the web services, class loading,
> > > Frequency of change: Routine.
> > >
> > > - Utilities: Things used by both the core and the extensions
> > > Frequency of change: Routine
> > >
> > > - Extensions: Processors, Prioritizers, Custom UIs, Reporting Tasks,
> > > Controller tasks, etc... . These are the things we want folks to be
> able
> > to
> > > build - the intentional points of extension.
> > > Frequency of change: Constant
> > >
> > > As I think through these different concerns and how maven works it
> isn't
> > > clear to me what a better model would be.  Having not yet gone through
> > the
> > > challenges of an official apache release I don''t appreciate yet what
> > pain
> > > points that brings either.
> > >
> > > Our current model requires someone to know a lot of ordering for the
> > > build.  But if we put a CI server behind it that would be less of an
> > issue
> > > except for those few folks that want to build the whole thing by hand.
> > For
> > > that perhaps the build-order is sufficient.
> > >
> > > So - we leaving it as is or is there a good proposal?
> > >
> > > Thanks
> > > Joe
> >
> >
> >
> > --
> > Joey Echeverria
> >
>
>
>
> --
> Sean
>

Re: resolution on the build / project source structure

[Sean Busbey <bu...@cloudera.com>]

Not to get into minutiae too much, but you can have a common parent pom and
still version / release independently. As an example, many (most?) ASF
projects use the Apache root pom as their parent, and that changes
infrequently. That said, I think the build mechanics part of the issue
isn't as important as the varying release cycles. We won't be the first ASF
project to use different builds for different components and a unifying
build script or makefile. (though I don't recall seeing one where those
different component builds are all actually maven)

I think the big cost for keeping the releases independent will be the
overhead of running all those release VOTEs, since you need to have at
least 3 PMCs sign off on release correctness (independent of the
correctness of the software in the release). It might be worth looking at
how things work over in Apache Commons, since they have a bunch of
differently versioned components.

The most important bit is that incubator is the time to figure this sort of
thing out. There's little risk in trying to maintain the current habit of
different release cycles. If it gets to be too much of a burden, we can
just adapt later. We could always adopt a versioning scheme that provides
different levels of compatibility guarantees for different kinds of
components. For example, in Avro's X.Y.Z versioning, X only increments if
data stored to disk can't be backwards compatibly read and Y increments for
breaking changes in the language specific implementation APIs.



On Fri, Dec 12, 2014 at 12:10 PM, Joey Echeverria <jo...@cloudera.com> wrote:

> When you say "Frequency of change", does that just refer to the level
> of development activity or do you expect different components to have
> different release cycles?
>
> I'm assuming the reason there isn't currently a parent POM with each
> of those just maven modules under the parent is because of the desire
> to version/release them independently.
>
> -Joey
>
> On Fri, Dec 12, 2014 at 6:37 AM, Joe Witt <jo...@gmail.com> wrote:
> > Folks,
> >
> > We need to come to a conclusion on what we'll do about the build process
> > specifically as it relates to the source structure - if anything.  The
> > longer we wait the more disruptive and confusing it will be especially
> for
> > potential new contributors.
> >
> > Background / Components / Points of tension:
> > - The 'Nifi API': This is a very small library which defines the
> interfaces
> > and some very basic abstract implementations or highly common functions.
> > The API is critical to building extensions and critical to the function
> of
> > the core application.
> > Frequency of change: Should be rare
> >
> > - The core application: This is the implementation of the data flow
> engine,
> > the web services, class loading,
> > Frequency of change: Routine.
> >
> > - Utilities: Things used by both the core and the extensions
> > Frequency of change: Routine
> >
> > - Extensions: Processors, Prioritizers, Custom UIs, Reporting Tasks,
> > Controller tasks, etc... . These are the things we want folks to be able
> to
> > build - the intentional points of extension.
> > Frequency of change: Constant
> >
> > As I think through these different concerns and how maven works it isn't
> > clear to me what a better model would be.  Having not yet gone through
> the
> > challenges of an official apache release I don''t appreciate yet what
> pain
> > points that brings either.
> >
> > Our current model requires someone to know a lot of ordering for the
> > build.  But if we put a CI server behind it that would be less of an
> issue
> > except for those few folks that want to build the whole thing by hand.
> For
> > that perhaps the build-order is sufficient.
> >
> > So - we leaving it as is or is there a good proposal?
> >
> > Thanks
> > Joe
>
>
>
> --
> Joey Echeverria
>



-- 
Sean

Re: resolution on the build / project source structure

[Joey Echeverria <jo...@cloudera.com>]

When you say "Frequency of change", does that just refer to the level
of development activity or do you expect different components to have
different release cycles?

I'm assuming the reason there isn't currently a parent POM with each
of those just maven modules under the parent is because of the desire
to version/release them independently.

-Joey

On Fri, Dec 12, 2014 at 6:37 AM, Joe Witt <jo...@gmail.com> wrote:
> Folks,
>
> We need to come to a conclusion on what we'll do about the build process
> specifically as it relates to the source structure - if anything.  The
> longer we wait the more disruptive and confusing it will be especially for
> potential new contributors.
>
> Background / Components / Points of tension:
> - The 'Nifi API': This is a very small library which defines the interfaces
> and some very basic abstract implementations or highly common functions.
> The API is critical to building extensions and critical to the function of
> the core application.
> Frequency of change: Should be rare
>
> - The core application: This is the implementation of the data flow engine,
> the web services, class loading,
> Frequency of change: Routine.
>
> - Utilities: Things used by both the core and the extensions
> Frequency of change: Routine
>
> - Extensions: Processors, Prioritizers, Custom UIs, Reporting Tasks,
> Controller tasks, etc... . These are the things we want folks to be able to
> build - the intentional points of extension.
> Frequency of change: Constant
>
> As I think through these different concerns and how maven works it isn't
> clear to me what a better model would be.  Having not yet gone through the
> challenges of an official apache release I don''t appreciate yet what pain
> points that brings either.
>
> Our current model requires someone to know a lot of ordering for the
> build.  But if we put a CI server behind it that would be less of an issue
> except for those few folks that want to build the whole thing by hand.  For
> that perhaps the build-order is sufficient.
>
> So - we leaving it as is or is there a good proposal?
>
> Thanks
> Joe



-- 
Joey Echeverria