build system

[Rafael Schloming <ra...@redhat.com>]

Below is a summary of the build system issues discussed at the Qpid
Java F2F in December:

  1) maven seems to suffer from unrepeatable builds, specifically:
    - consecutive offline builds sometimes produce different results

  2) maven is unsuitable for OS distro builds, specifically:
    - no true offline build capability
    - can't pick up dependencies from standard filesystem locations

  3) mvn clean doesn't clean up everything

  4) javadoc doesn't work

  5) there is no easy way to run the broker/examples/etc directly out
     of a source build

  6) we need to manually create poms for client.jar

Issues 1 and 2 are general problems with maven. Issues 3 and 4 are
probably peculiar to our usage of maven. Issue 5 is somewhere
in-between, i.e. this could probably be made to work with maven,
however all other build systems seem to provide this sort of thing
"for free."

The suggested resolution from the F2F is to switch back to the ant
based build system used prior to maven and enhance it to automatically
produce the necessary poms for downstream maven projects to consume
our releases.

In addition to addressing issues 1 and 2, this would address issues 3,
4, and 5 "for free" since those items are already supported in the ant
build system.

If this plan is agreed to, I would volunteer to do the work of
tesurrecting the ant build system and enhancing it to produce the
necessary poms along with the regular release artifcats.

If you were at the F2F, please jump in if I've overlooked any
important points. Others please comment on the issues, suggested
resolution, timing, etc. If there is consensus I'd like to move
forward sooner rather than later.

--Rafael

Re: build system

[Rajith Attapattu <ra...@gmail.com>]

Rafi,

I agree with most of what is stated below.
Here is my +1 to move to ant.

Regards,

Rajith

On Jan 8, 2008 3:53 PM, Rafael Schloming <ra...@redhat.com> wrote:

> Below is a summary of the build system issues discussed at the Qpid
> Java F2F in December:
>
>  1) maven seems to suffer from unrepeatable builds, specifically:
>    - consecutive offline builds sometimes produce different results
>
>  2) maven is unsuitable for OS distro builds, specifically:
>    - no true offline build capability
>    - can't pick up dependencies from standard filesystem locations
>
>  3) mvn clean doesn't clean up everything
>
>  4) javadoc doesn't work
>
>  5) there is no easy way to run the broker/examples/etc directly out
>     of a source build
>
>  6) we need to manually create poms for client.jar
>
> Issues 1 and 2 are general problems with maven. Issues 3 and 4 are
> probably peculiar to our usage of maven. Issue 5 is somewhere
> in-between, i.e. this could probably be made to work with maven,
> however all other build systems seem to provide this sort of thing
> "for free."
>
> The suggested resolution from the F2F is to switch back to the ant
> based build system used prior to maven and enhance it to automatically
> produce the necessary poms for downstream maven projects to consume
> our releases.
>
> In addition to addressing issues 1 and 2, this would address issues 3,
> 4, and 5 "for free" since those items are already supported in the ant
> build system.
>
> If this plan is agreed to, I would volunteer to do the work of
> tesurrecting the ant build system and enhancing it to produce the
> necessary poms along with the regular release artifcats.
>
> If you were at the F2F, please jump in if I've overlooked any
> important points. Others please comment on the issues, suggested
> resolution, timing, etc. If there is consensus I'd like to move
> forward sooner rather than later.
>
> --Rafael
>
>


-- 
Regards,

Rajith Attapattu
Red Hat
blog: http://rajith.2rlabs.com/

Re: build system

[Martin Ritchie <ri...@apache.org>]

On 09/01/2008, Rupert Smith <ru...@googlemail.com> wrote:
> 1) maven seems to suffer from unrepeatable builds, specifically:
>
> We do manage to use it repeatedly by preserving all dependencies when doing
> a release build. Do an on-line build to suck in all dependencies, then an
> off-line against the dependencies. All dependencies get preserved in an
> internal repository.
>
>  2) maven is unsuitable for OS distro builds, specifically:
>    - no true off-line build capability
>
> Yup, this seems to be a problem for Linux users. True off-line does work
> though, I build maven projects on my laptop when there is no net connection
> and it works just fine.
>
>    - can't pick up dependencies from standard filesystem locations.
>
> Can, but not if you want to release into central repo, as your file system
> is only visible locally. To participate in the maven Java OS ecosystem you
> need to only include dependencies that are also available in the central
> repo.
>
>  3) mvn clean doesn't clean up everything
>
> Does if you put all generated artefacts under /target (the maven way...).
>
>  4) javadoc doesn't work
>
> Mysterious, but I'm sure it can be made to. Certainly I have it working on
> other M2 projects.
>
>  5) there is no easy way to run the broker/examples/etc directly out
>     of a source build
>
> Not sure what you mean by this? Can't exec an external process? Can do that
> by injecting a little bit of ant into the maven build script.
>
>  6) we need to manually create poms for client.jar
>
> Also a bit ridiculous. Maven project ought to be able to correctly generate
> its poms, we are clearly using it in an unusual fashion if not. Again, I
> have this sort of thing working perfectly on other maven projects.
>
> I think only 2a is a genuine issue for linux users, and those wanting to
> build Java packages for linux distributions.
>
> I think the answer as to why we want to use Ant and not Maven or Buildr, is
> one of familiarity. There is a general consensus that those contributing to
> the Java are more familiar and comfortable with Ant and would see adoption
> of Ant as enabling them to get on with writing the software than wrestling
> with Maven. Which is fine, I have absolutely no problems with that. The real
> issue is that maven has a learning curve that most of us are not willing to
> climb (and some mysterious behaviours too).
>
> This would also be the reason for not attempting to use Buildr. It might be
> the latest thing, but its one more thing to learn, and none of us can
> guarantee that the end result will be any better.
>
> We should make a list of things that the current build system does that its
> replacement should also do. Here are the ones I can think of (aside from the
> obvious ones of building the .jars and .zips and .tar.gzs):
>
> Build a retro translated Java 1.4 version of the client.
> Create poms for all broker and client build artefacts, and suitably named
> and version stamped build artefacts, for inclusion in a maven 2 repository
> (a template pom and a little XSLT should do the trick from Ant).
> Perform builds that depend an all tests passing.
> Generate test scripts for performance tests, and package these tests as a
> distribution.
> Run checkstyle against the code.
> Run clover (or equivalent) against the tests.
> Include local properties for default log4j.configurations (so that each
> developer can use a local custom set up for it).
> Inject subversion revision numbers, project and module names into each .jar
> on every build, so that the source version for 'lost' jars can always be
> identified.
> Be able to start/stop an external broker to run tests against.
> Run the python tests against an external broker.
>
> Can anyone think of anything else?

We shouldn't also forget we created a bunch of Jiras relating to
features that the ant system had that maven didn't do when we started
using maven. AFAIR they were not all implemented so we should ensure
we include that functionality in any future build system

> Rupert
>
>
> On 09/01/2008, Robert Greig <ro...@gmail.com> wrote:
> >
> > On 09/01/2008, Rafael Schloming <ra...@redhat.com> wrote:
> >
> > > We did briefly discuss this option. I believe the primary objection to
> > > buildr was from windows users who wished to avoid introducing a ruby
> > > dependency to the Java build.
> >
> > OK, that makes sense.
> >
> > RG
> >
>


-- 
Martin Ritchie

Re: build system

[Rupert Smith <ru...@googlemail.com>]

1) maven seems to suffer from unrepeatable builds, specifically:

We do manage to use it repeatedly by preserving all dependencies when doing
a release build. Do an on-line build to suck in all dependencies, then an
off-line against the dependencies. All dependencies get preserved in an
internal repository.

 2) maven is unsuitable for OS distro builds, specifically:
   - no true off-line build capability

Yup, this seems to be a problem for Linux users. True off-line does work
though, I build maven projects on my laptop when there is no net connection
and it works just fine.

   - can't pick up dependencies from standard filesystem locations.

Can, but not if you want to release into central repo, as your file system
is only visible locally. To participate in the maven Java OS ecosystem you
need to only include dependencies that are also available in the central
repo.

 3) mvn clean doesn't clean up everything

Does if you put all generated artefacts under /target (the maven way...).

 4) javadoc doesn't work

Mysterious, but I'm sure it can be made to. Certainly I have it working on
other M2 projects.

 5) there is no easy way to run the broker/examples/etc directly out
    of a source build

Not sure what you mean by this? Can't exec an external process? Can do that
by injecting a little bit of ant into the maven build script.

 6) we need to manually create poms for client.jar

Also a bit ridiculous. Maven project ought to be able to correctly generate
its poms, we are clearly using it in an unusual fashion if not. Again, I
have this sort of thing working perfectly on other maven projects.

I think only 2a is a genuine issue for linux users, and those wanting to
build Java packages for linux distributions.

I think the answer as to why we want to use Ant and not Maven or Buildr, is
one of familiarity. There is a general consensus that those contributing to
the Java are more familiar and comfortable with Ant and would see adoption
of Ant as enabling them to get on with writing the software than wrestling
with Maven. Which is fine, I have absolutely no problems with that. The real
issue is that maven has a learning curve that most of us are not willing to
climb (and some mysterious behaviours too).

This would also be the reason for not attempting to use Buildr. It might be
the latest thing, but its one more thing to learn, and none of us can
guarantee that the end result will be any better.

We should make a list of things that the current build system does that its
replacement should also do. Here are the ones I can think of (aside from the
obvious ones of building the .jars and .zips and .tar.gzs):

Build a retro translated Java 1.4 version of the client.
Create poms for all broker and client build artefacts, and suitably named
and version stamped build artefacts, for inclusion in a maven 2 repository
(a template pom and a little XSLT should do the trick from Ant).
Perform builds that depend an all tests passing.
Generate test scripts for performance tests, and package these tests as a
distribution.
Run checkstyle against the code.
Run clover (or equivalent) against the tests.
Include local properties for default log4j.configurations (so that each
developer can use a local custom set up for it).
Inject subversion revision numbers, project and module names into each .jar
on every build, so that the source version for 'lost' jars can always be
identified.
Be able to start/stop an external broker to run tests against.
Run the python tests against an external broker.

Can anyone think of anything else?

Rupert


On 09/01/2008, Robert Greig <ro...@gmail.com> wrote:
>
> On 09/01/2008, Rafael Schloming <ra...@redhat.com> wrote:
>
> > We did briefly discuss this option. I believe the primary objection to
> > buildr was from windows users who wished to avoid introducing a ruby
> > dependency to the Java build.
>
> OK, that makes sense.
>
> RG
>

Re: build system

[Robert Greig <ro...@gmail.com>]

On 09/01/2008, Rafael Schloming <ra...@redhat.com> wrote:

> We did briefly discuss this option. I believe the primary objection to
> buildr was from windows users who wished to avoid introducing a ruby
> dependency to the Java build.

OK, that makes sense.

RG

Re: build system

[Rafael Schloming <ra...@redhat.com>]

Robert Greig wrote:
> On 08/01/2008, Rafael Schloming <ra...@redhat.com> wrote:
> 
>> The suggested resolution from the F2F is to switch back to the ant
>> based build system used prior to maven and enhance it to automatically
>> produce the necessary poms for downstream maven projects to consume
>> our releases.
>>
>> In addition to addressing issues 1 and 2, this would address issues 3,
>> 4, and 5 "for free" since those items are already supported in the ant
>> build system.
>>
>> If this plan is agreed to, I would volunteer to do the work of
>> tesurrecting the ant build system and enhancing it to produce the
>> necessary poms along with the regular release artifcats.
>>
>> If you were at the F2F, please jump in if I've overlooked any
>> important points. Others please comment on the issues, suggested
>> resolution, timing, etc. If there is consensus I'd like to move
>> forward sooner rather than later.
> 
> I am comfortable with this, particularly since I know the ant build
> system we had was very good.
> 
> For the record, for anyone looking back at this in future, did you
> consider any other options at the F2F such as Apache Buildr? If so can
> you list out the reasons for rejection in favour of ant?

We did briefly discuss this option. I believe the primary objection to 
buildr was from windows users who wished to avoid introducing a ruby 
dependency to the Java build.

--Rafael

Re: build system

[Robert Greig <ro...@gmail.com>]

On 08/01/2008, Rafael Schloming <ra...@redhat.com> wrote:

> The suggested resolution from the F2F is to switch back to the ant
> based build system used prior to maven and enhance it to automatically
> produce the necessary poms for downstream maven projects to consume
> our releases.
>
> In addition to addressing issues 1 and 2, this would address issues 3,
> 4, and 5 "for free" since those items are already supported in the ant
> build system.
>
> If this plan is agreed to, I would volunteer to do the work of
> tesurrecting the ant build system and enhancing it to produce the
> necessary poms along with the regular release artifcats.
>
> If you were at the F2F, please jump in if I've overlooked any
> important points. Others please comment on the issues, suggested
> resolution, timing, etc. If there is consensus I'd like to move
> forward sooner rather than later.

I am comfortable with this, particularly since I know the ant build
system we had was very good.

For the record, for anyone looking back at this in future, did you
consider any other options at the F2F such as Apache Buildr? If so can
you list out the reasons for rejection in favour of ant?

RG