Release process in POMs

[Benson Margulies <bi...@gmail.com>]

I spent the last few days skating a bit close to the edge in fighting
with the POMs to get the release to stick together.

A few notes on the carnage, and a suggesting for the next time around...

I pitched the use of org:apache:7 as our parent. It's advertised as a
generally sensible parent pom for all Apache maven projects, but I
feel, pretty vehemently, that this is false advertising. I know of one
other very experienced PMC chair of a large Apache project in mvn who
feels the same way. So it's gone. If the maven gang every makes a
parent pom that sticks to providing what's useful and doesn't include
intrusive and unwanted settings, this could be reconsidered.

I'm striving to rationalize the work that happens during :prepare and
the work that happens during :perform. Now that we don't have a gnu
pointed to our heads, I'd really like to know what other people think.

Keep in mind that the purpose of perform is to create a tag pointing
to a version with the proposed release numbers in the poms, and the
purpose of prepare is to checkout that tag, build, and push.

The release plugin has an easy way to activate profiles for perform,
and no special theory about prepare.

Here's a proposal, what do folks think?

NORMAL DEVELOPER RUNS MVN:

* full unit tests
* no distribution packages
* example job

NORMAL DEVELOPER RUNS MVN -Peverything: (useful for things like mvn
versions:set)

* full unit tests
* mvn includes distribution directory, but distribution directory does
not actually build distribution
* example job

NORMAL DEVELOPER RUNS MVN -Peverything,release (useful for testing
release wiring)
 * full unit tests
 * full distribution package
 * example job

RM RUNS MVN release:prepare -Peverything,fastinstall
 * no unit tests
 * mvn visits distribution to create tag, but does not build bundles
 * no example job

RM RUNS MVN release:perform

  *no unit tests
  *full distribution
  *example job

Re: Release process in POMs

[Ted Dunning <te...@gmail.com>]

THanks.  I still don't necessarily get it all, but it is making a bit more
sense.  You comments about
including all modules strikes home.

On Sun, Oct 31, 2010 at 2:10 PM, Benson Margulies <bi...@gmail.com>wrote:

> On Sun, Oct 31, 2010 at 4:42 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > On Sun, Oct 31, 2010 at 9:27 AM, Benson Margulies <bimargulies@gmail.com
> >wrote:
> >
> >> I'm striving to rationalize the work that happens during :prepare and
> >> the work that happens during :perform. Now that we don't have a gnu
> >> pointed to our heads, I'd really like to know what other people think.
> >>
> >
> > I like the typo.
>
> I confess. It was on purpose.
> >
> >
> >> Keep in mind that the purpose of perform is to create a tag pointing
> >> to a version with the proposed release numbers in the poms, and the
> >> purpose of prepare is to checkout that tag, build, and push.
> >>
> >
> > Are perform/prepare backwards here?
>
> Yes.
>
> >
> >
> >> Here's a proposal, what do folks think?
> >>
> >> NORMAL DEVELOPER RUNS MVN:
> >>
> >> * full unit tests
> >> * no distribution packages
> >> * example job
> >>
> >
> > Yes.  If would even be nice to have a way to disable the example job much
> > the way that the unit tests can be disabled.
> >
> > It would be nice to build the normal mahout jars here in addition to the
> > example job.  This is because lots of Mahout things
> > are done locally or via the Mahout API.  Having the jar handy is nice.
> >
> >
> >> NORMAL DEVELOPER RUNS MVN -Peverything: (useful for things like mvn
> >> versions:set)
> >>
> >> * full unit tests
> >> * mvn includes distribution directory, but distribution directory does
> >> not actually build distribution
> >> * example job
> >>
> >
> > I don't understand the rationale here.
> >
> > How do you decide what to do in this case?  Also, what does "includes
> > distribution directory" really mean?
>
> In answering your question, I realize that I think we have a
> fundamental error in the top-level POM, so let me explain here what we
> should have instead.
>
> As the top-level POM exists today, the <module>distribution</module>
> element is not in the base POM. It's only added in by profiles. I have
> learned by bitter experience that this is a really bad idea. There are
> many useful maven plugins that traverse the tree of children.
> release:prepare is one. versions:set is another. Having a module that
> 'hides' in a profile is very error prone. The correct design is to
> include the distribution module unconditionally, and control the
> amount of work we do in the distribution directory via profile control
> of the 'skip' parameter of the plugins in it, or by just putting the
> plugin executions that do real work in a profile. So please assume
> that a next revision of this idea, possibly to move to a JIRA, will
> remove the entire concept of an 'everything' profile.
>
>
> >
> >
> >> NORMAL DEVELOPER RUNS MVN -Peverything,release (useful for testing
> >> release wiring)
> >>  * full unit tests
> >>  * full distribution package
> >>  * example job
> >>
> >
> > This is a critical use case, but how does somebody know that
> > -Peverything,release is the right thing to use here?
> > Is this Mahout specific or general mvn convention?
>
> Forget about 'everything'. Naming a profile 'release' that is used to
> turn on full distribution processing is very typical, not
> Mahout-specific one bit.
>
> >
> >
> >> RM RUNS MVN release:prepare -Peverything,fastinstall
> >>  * no unit tests
> >>  * mvn visits distribution to create tag, but does not build bundles
> >>  * no example job
> >>
> >
> > What is the profile doing for us here?  Is there some sensible use case
> > where release:prepare without these profiles?  If not, why
> > can't the profiles be wired into the release:prepare goal?
>
> We can't wire profiles into release:prepare because of how it works.
>
> Here's what's going on. 'perform' forks a whole new copy of maven, in
> a whole new process, to run a build in the new checkout. So, it can
> control the active profiles, and it has a parameter that allows you to
> specify which profiles to activate.
>
> 'Prepare' just runs right now. It has to put up with whatever profiles
> you happen to have activated.
>
> However, we could restore the 'release_prepare' profile that i
> eliminated (oops) and put the execution of prepare into there. Thus,
> our local convention would NOT be 'mvn release:prepare', but rather
> 'mvn -Prelease_prepare'.
>
> >
> > What does "visits distribution to create tag" specifically mean?  I think
> I
> > understand "create tag" to mean an SVN operation, but
> > what does it mean in the context of "visits distribution"?
>
> See my screed above. At 'prepare', we need ALL of the poms to get
> edited and checked in with their new versions, so that the version in
> the tag will have the right things. If 'distribution' isn't traversed,
> because it isn't in the module list, it won't get processed.
>
>
> >
> >
> >> RM RUNS MVN release:perform
> >>
> >>  *no unit tests
> >>  *full distribution
> >>  *example job
> >>
> >
> > makes sense.
> >
> > Does this push to nexus as well?  Or is that manual?
>
> Yes. It pushes to Nexus.
>
> >
>

Re: Release process in POMs

[Benson Margulies <bi...@gmail.com>]

On Sun, Oct 31, 2010 at 4:42 PM, Ted Dunning <te...@gmail.com> wrote:
> On Sun, Oct 31, 2010 at 9:27 AM, Benson Margulies <bi...@gmail.com>wrote:
>
>> I'm striving to rationalize the work that happens during :prepare and
>> the work that happens during :perform. Now that we don't have a gnu
>> pointed to our heads, I'd really like to know what other people think.
>>
>
> I like the typo.

I confess. It was on purpose.
>
>
>> Keep in mind that the purpose of perform is to create a tag pointing
>> to a version with the proposed release numbers in the poms, and the
>> purpose of prepare is to checkout that tag, build, and push.
>>
>
> Are perform/prepare backwards here?

Yes.

>
>
>> Here's a proposal, what do folks think?
>>
>> NORMAL DEVELOPER RUNS MVN:
>>
>> * full unit tests
>> * no distribution packages
>> * example job
>>
>
> Yes.  If would even be nice to have a way to disable the example job much
> the way that the unit tests can be disabled.
>
> It would be nice to build the normal mahout jars here in addition to the
> example job.  This is because lots of Mahout things
> are done locally or via the Mahout API.  Having the jar handy is nice.
>
>
>> NORMAL DEVELOPER RUNS MVN -Peverything: (useful for things like mvn
>> versions:set)
>>
>> * full unit tests
>> * mvn includes distribution directory, but distribution directory does
>> not actually build distribution
>> * example job
>>
>
> I don't understand the rationale here.
>
> How do you decide what to do in this case?  Also, what does "includes
> distribution directory" really mean?

In answering your question, I realize that I think we have a
fundamental error in the top-level POM, so let me explain here what we
should have instead.

As the top-level POM exists today, the <module>distribution</module>
element is not in the base POM. It's only added in by profiles. I have
learned by bitter experience that this is a really bad idea. There are
many useful maven plugins that traverse the tree of children.
release:prepare is one. versions:set is another. Having a module that
'hides' in a profile is very error prone. The correct design is to
include the distribution module unconditionally, and control the
amount of work we do in the distribution directory via profile control
of the 'skip' parameter of the plugins in it, or by just putting the
plugin executions that do real work in a profile. So please assume
that a next revision of this idea, possibly to move to a JIRA, will
remove the entire concept of an 'everything' profile.


>
>
>> NORMAL DEVELOPER RUNS MVN -Peverything,release (useful for testing
>> release wiring)
>>  * full unit tests
>>  * full distribution package
>>  * example job
>>
>
> This is a critical use case, but how does somebody know that
> -Peverything,release is the right thing to use here?
> Is this Mahout specific or general mvn convention?

Forget about 'everything'. Naming a profile 'release' that is used to
turn on full distribution processing is very typical, not
Mahout-specific one bit.

>
>
>> RM RUNS MVN release:prepare -Peverything,fastinstall
>>  * no unit tests
>>  * mvn visits distribution to create tag, but does not build bundles
>>  * no example job
>>
>
> What is the profile doing for us here?  Is there some sensible use case
> where release:prepare without these profiles?  If not, why
> can't the profiles be wired into the release:prepare goal?

We can't wire profiles into release:prepare because of how it works.

Here's what's going on. 'perform' forks a whole new copy of maven, in
a whole new process, to run a build in the new checkout. So, it can
control the active profiles, and it has a parameter that allows you to
specify which profiles to activate.

'Prepare' just runs right now. It has to put up with whatever profiles
you happen to have activated.

However, we could restore the 'release_prepare' profile that i
eliminated (oops) and put the execution of prepare into there. Thus,
our local convention would NOT be 'mvn release:prepare', but rather
'mvn -Prelease_prepare'.

>
> What does "visits distribution to create tag" specifically mean?  I think I
> understand "create tag" to mean an SVN operation, but
> what does it mean in the context of "visits distribution"?

See my screed above. At 'prepare', we need ALL of the poms to get
edited and checked in with their new versions, so that the version in
the tag will have the right things. If 'distribution' isn't traversed,
because it isn't in the module list, it won't get processed.


>
>
>> RM RUNS MVN release:perform
>>
>>  *no unit tests
>>  *full distribution
>>  *example job
>>
>
> makes sense.
>
> Does this push to nexus as well?  Or is that manual?

Yes. It pushes to Nexus.

>

Re: Release process in POMs

[Ted Dunning <te...@gmail.com>]

On Sun, Oct 31, 2010 at 9:27 AM, Benson Margulies <bi...@gmail.com>wrote:

> I'm striving to rationalize the work that happens during :prepare and
> the work that happens during :perform. Now that we don't have a gnu
> pointed to our heads, I'd really like to know what other people think.
>

I like the typo.

My general feeling is that even with documentation, the process that
happened was waay too magical
for my test.  I think that the basic skeleton is there in a hazy kind of
way, but there were lots of things
that you fixed that I would have been unable to fathom.  I don't like that.


> Keep in mind that the purpose of perform is to create a tag pointing
> to a version with the proposed release numbers in the poms, and the
> purpose of prepare is to checkout that tag, build, and push.
>

Are perform/prepare backwards here?


> Here's a proposal, what do folks think?
>
> NORMAL DEVELOPER RUNS MVN:
>
> * full unit tests
> * no distribution packages
> * example job
>

Yes.  If would even be nice to have a way to disable the example job much
the way that the unit tests can be disabled.

It would be nice to build the normal mahout jars here in addition to the
example job.  This is because lots of Mahout things
are done locally or via the Mahout API.  Having the jar handy is nice.


> NORMAL DEVELOPER RUNS MVN -Peverything: (useful for things like mvn
> versions:set)
>
> * full unit tests
> * mvn includes distribution directory, but distribution directory does
> not actually build distribution
> * example job
>

I don't understand the rationale here.

How do you decide what to do in this case?  Also, what does "includes
distribution directory" really mean?


> NORMAL DEVELOPER RUNS MVN -Peverything,release (useful for testing
> release wiring)
>  * full unit tests
>  * full distribution package
>  * example job
>

This is a critical use case, but how does somebody know that
-Peverything,release is the right thing to use here?
Is this Mahout specific or general mvn convention?


> RM RUNS MVN release:prepare -Peverything,fastinstall
>  * no unit tests
>  * mvn visits distribution to create tag, but does not build bundles
>  * no example job
>

What is the profile doing for us here?  Is there some sensible use case
where release:prepare without these profiles?  If not, why
can't the profiles be wired into the release:prepare goal?

What does "visits distribution to create tag" specifically mean?  I think I
understand "create tag" to mean an SVN operation, but
what does it mean in the context of "visits distribution"?


> RM RUNS MVN release:perform
>
>  *no unit tests
>  *full distribution
>  *example job
>

makes sense.

Does this push to nexus as well?  Or is that manual?