Building and testing with snapshot dependencies, new Jenkins build

[Holly Cummins <ho...@googlemail.com>]

A while ago we had several discussions about what our policy for
dependency versions should be and how we ensured adequate test
coverage (http://mail-archives.apache.org/mod_mbox/aries-dev/201203.mbox/%3CCAC9tjn9-_MPKA5NAPwLOq3kMqJPh1O=Jwsi-8yTLMy72exMViA@mail.gmail.com%3E).
I proposed the following:

Phase 1: We build twice in Jenkins, using the maven versions plugin
(with an include list of org.apache.aries.*) to ensure we build both a
slice of current code, and with the minimum versions.
Phase 2: We enable developers to also test twice, to avoid the
possibility of committing regressions and breaking Jenkins.
Phase 3: We make testing in both modes so transparent developers don't
have to do anything explicitly, and so that it also works in
non-command-line environments like Eclipse

I've now completed Phase 1, so we now have two Aries builds,

https://builds.apache.org/job/Aries/
https://builds.apache.org/job/AriesWithSnapshotDependencies/

The Aries build takes our source tree and builds it to produce
releasable artefacts. The Aries with snapshot dependencies build
manipulates all the poms so that all dependencies are replaced with
the latest snapshots. This gives us test coverage of a 'slice' of all
the current functionality working together.

HOW DO I KNOW WHAT THE SNAPSHOT BUILD IS BUILDING?

If you want to double check that dependencies were updated, or see
what was actually tested, the modified poms are available as build
artefacts. For example, to see the pom which was used for the proxy
itests, the link is:

https://builds.apache.org/job/AriesWithSnapshotDependencies/205/org.apache.aries.proxy$org.apache.aries.proxy.itests/artifact/org.apache.aries.proxy/org.apache.aries.proxy.itests/1.0.0-SNAPSHOT/org.apache.aries.proxy.itests-1.0.0-SNAPSHOT.pom

WHAT HAPPENS IF THE SNAPSHOT BUILD FAILS?

We may see failures in the snapshot build which aren't reproducible in
trunk. They can be reproduced by pulling down the relevant poms from
the build (but being super-careful not to commit them back to trunk.)
I'm working on Phase 2, which should make reproducing anything much
easier.


WHY IS THIS NECESSARY?

Bundles have a contract about what versions of their semantically
versioned dependencies they work with - they work with both the bottom
of the imported range and the top. In order to test both ends of the
range, it's really necessary to run tests twice.

What's tested in trunk should be the bottom end of the range, for
several reasons.

For example, say we have two modules, foo.api and foo.impl. For
simplicity, the foo.api bundle has just one package, foo.api. The
foo.impl pom has a dependency on foo.api. If a new method is added to
foo.api, foo.api's pom and package version should change from 1.0.0 to
1.1.0 As a provider of foo.api, foo.impl will need to depend on
version 1.1.0 of the foo.api pom - otherwise we'll get compile
failures (assuming we have @override annotations).

What happens if instead we just correct a typo in a message in
foo.api? The pom  and package version should change from 1.0.0 to
1.0.1. The dependency in the foo.impl pom *must* stay at 1.0.0. If
foo.impl depends on foo.api 1.0.1, foo.impl won't resolve in a system
where the foo.api package is available at 1.0.0, even though the only
difference is a typo correction. Once foo.impl is changed to depend on
foo.api 1.0.1, the information about whether we think foo.impl would
work with foo.api 1.0.0 is lost forever, so we need to reflect the
minimum version of our dependencies we expect to work with in our
poms.

"Yes, but then how do I ensure foo.impl actually works with foo.api
1.0.1? I can't be writing code and never exercising it!" I hear you
ask. The good news is that while it's very difficult to work out the
minimum version of a dependency we'll tolerate, if it's not written
down in a pom, it's trivial to test against the maximum version of a
dependency.

I'll find a suitable web page and update it with details of the builds.

Holly