You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@gump.apache.org by "Mark R. Diggory" <md...@latte.harvard.edu> on 2004/01/29 21:21:01 UTC
Nightly builds, Gump, Maven and repository issues.
Hi all,
I wanted to to attempt to coordinate with Gump folks concerning nightly
builds, the new maven repository on the apache server and other
interesting tidbits.
1.) Maven Repository on minotaur.
We migrated the maven repository contents being created by Apache
projects over to minotaur, they are currently under
/www/www.apache.org/dist/java-repository
this is mirrored and ibiblio.org is working to sync it up to the
canonical maven repository there.
We're not 100% sure this will be a permantent location, I've been
attempting to integrate it into the dist directory contents. It
currently holds both dated snapshot builds and versioned release builds
of jar files for many of the Apache projects.
2.) There is the nightly build process on minotaur that Craig McClanahan
runs. This generates nightly builds under
/www/cvs.apache.org/builds/...
This produces tar and zip archives of the various distributions
registered to be processed with it.
Mandates concerning www.apache.org/dist are to have it contain versioned
releases. As such, I started considering spliting the Maven contents
into two places:
for releases:
/www/www.apache.org/dist/java-repository
for developer/testing/nightly builds:
/www/cvs.apache.org/builds/java-repository
The former will be mirrored and synced into ibiblio.org/maven.
The later will be accessable to Apache developers for dependency testing
and development.
3.) There are the various continuous integration builds gump is
managing. These are producing jars and other distributables. It would be
beneficial to see these various sources unified or synced up in someway.
----
So in an attempt to get more "consistent" in terms of storing contents
of these various efforts, what are your thoughts on
cooperation/consolidation of the contents of these various build processes?
The goal on my end is to promote a nightly build process which helps
populate the two proposed repository locations.
thanks for any input,
-Mark
--
Mark Diggory
Jakarta Commons Math Project
Re: Nightly builds, Gump, Maven and repository issues.
Posted by Nick Chalko <ni...@chalko.com>.
When I used gump to do centipede nightly builds, the key was to set
<property name="build.sysclasspath" value="last"/>
instead of
<property name="build.sysclasspath" value="only"/>
R,
Nick
Re: Nightly builds, Gump, Maven and repository issues.
Posted by Stefan Bodewig <bo...@apache.org>.
On Fri, 30 Jan 2004, Mark R. Diggory <md...@latte.harvard.edu>
wrote:
> The best example I can give of developer requirements this scenario
> is the following:
I think I understood that when I responded - and I think you
understood what Gump did 8-)
> The way I have it configured, I do all my development against either
> "dated version snapshot builds" that are archived in the maven
> repository.[snip] or actual released versions when cutting a
> release:
In Gump's case the "dated version snapshot builds" are less controlled
- and this is where I expect problems to come up.
commons-collections-20040102 is compiled against commons-lang-20040102
and may not work with commons-lang-2.0 at all. This is a completely
fictuos example.
> Currently the production of these dated builds is not automated. Its
> up to each project to release snapshots into the maven repository.
I'm all for coordinating this.
And as I already said, Gump can publish the created jars right now,
it's all a matter of how we get them from say the covalent.net machine
(or a future gump.apache.org) into the directory layout you envision.
> 1.) In a daily build process for maven projects, we'ed want either
> Maven or the Ant build.xml that can be generated by maven to get
> called
you don't want to rule out projects that build with vanilla Ant, do
you ;-)
For all projects that build successfully in Gump except for
bootstrap-ant you can assume that they have a working build.xml that
creates the jars. Which targets Gump executes is completely under
control of the project - or whoever takes care of the metadata
describing it.
> This way the artifacts would be comperable for a project across the
> various build processes.
This is nothing Gump can enforce. Traditionally Gump has been picking
up the build files as they are, injected properties to its liking and
tried to figure out which target to invoke. I don't think you'll ever
get all projects built by Gump to follow any target name convention at
all.
Stefan
Re: Nightly builds, Gump, Maven and repository issues.
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
Adam R. B. Jack wrote:
> Mark wrote:
>
>
>>I wanted to to attempt to coordinate with Gump folks concerning nightly
>>builds, the new maven repository on the apache server and other
>>interesting tidbits.
>
>
> I needed some time to mull this over before I responded. I see you have good
> intentions to try to pull a bunch of things together, and I respect that.
> That said, I don't think they should fit.
>
> Every time I try to match Gump with nightly builds, I come up against a
> philosophical problem which Stefan explained. Gump isn't doing nightly
> builds (against a stable base), it is doing nightly integration test builds
> (an unstable 'box of chocolates' :). Gump's output jars aren't for general
> use (few folks could/should live on the edge that much, each morning getting
> a shifting base). IMHO they probably ought not have been distributed (in the
> past) but for the future I see them as key to scaling a cascade of Gumps. As
> such, although I think the Gump repository ought be open/available (no point
> thinking for users) I envisage it's use only by other downstream Gumps.
>
> regards,
>
> Adam
>
The best example I can give of developer requirements this scenario is
the following:
I work on the commons-math project, which is dependent on
commons-collections and commons-logging.
The way I have it configured, I do all my development against either
"dated version snapshot builds" that are archived in the maven repository.
commons-collections-20040102.233541.jar 29-Jan-2004 19:46 842k
commons-collections-20040102.233541.jar.md5 29-Jan-2004 19:46 1k
commons-collections-SNAPSHOT.jar 29-Jan-2004 19:46 842k
commons-collections-SNAPSHOT.jar.md5 29-Jan-2004 19:46 1k
commons-collections-snapshot-version 29-Jan-2004 19:46 1k
or actual released versions when cutting a release:
commons-collections-3.0.jar 25-Jan-2004 01:50 506k
commons-collections-3.0.jar.md5 26-Jan-2004 15:53 1k
Currently the production of these dated builds is not automated. Its up
to each project to release snapshots into the maven repository.
My position is that it would be nice to have this snapshot process
automated in a nightly build such that these were generated on a regular
basis. The most important feature I see that it would be difficult to
maintain, is that the jars produced have to maintain the same content
and tructure as those tht would be produced normally by maven as
configured in the projects project.xml If gumps jars are very different.
then that could be an issue.
My thoughts are along these lines
1.) In a daily build process for maven projects, we'ed want either Maven
or the Ant build.xml that can be generated by maven to get called (this
is actually what happens in Craig's nightly build process right now, an
Ant build.xml file is required in the proect with a "dist" target
avalaible in it, and maven can actually produce thsi properly). This way
the artifacts would be comperable for a project across the various build
processes.
2.) Integration testing is on a very different scale and distribution
than nightly or weekly build releasing. So it may be the case that these
approaches remain different processes simply because they run at
different times, duration and scale. But, theres alot to be gained
sharing the automation approaches applied and exploring areas of
consolidation or standardization in process.
-Mark
--
Mark Diggory
Software Developer
Harvard MIT Data Center
http://www.hmdc.harvard.edu
Re: Nightly builds, Gump, Maven and repository issues.
Posted by Leo Simons <le...@apache.org>.
Adam R. B. Jack wrote:
> I've been having my fingers crossed for a dedicated box to start exploring
> such things (it seems rude to experiment like this on loaned resources.)
you can do whatever you want on lsd (that you have access to do) as long
as everything is `nice`d (so webserving gets priority and my wiki stays
up) and bandwidth usage is not astronomical (50gb a week is okay, for
example). BTW, I added another 60GB harddrive recently so there's plenty
of space for now.
However, the more builds run on it, the bigger the chance someone kills
them off because they're annoying them. If people start complaining too
much, we'll just need a "Kill gump" icon on the desktop....consider it
all as just another social experiment :D
--
cheers,
- Leo Simons
-----------------------------------------------------------------------
Weblog -- http://leosimons.com/
IoC Component Glue -- http://jicarilla.org/
Articles & Opinions -- http://articles.leosimons.com/
-----------------------------------------------------------------------
"We started off trying to set up a small anarchist community, but
people wouldn't obey the rules."
-- Alan Bennett
Re: Nightly builds, Gump, Maven and repository issues.
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
> At the same time, I think it would be useful to have both things: each
> module should be built twice, the first time with "latest-greatest
> dependencies" and the second time with "tested dependencies".
I've been having my fingers crossed for a dedicated box to start exploring
such things (it seems rude to experiment like this on loaned resources.)
> This might require some thinking but it would definately solve the
> issues and yield two different signals: one for the box of chocolates,
> one for the nightly built process.
>
> I think they are both important.
I'd like to see the 'Gump Engine' used for a number of things related to
pulling communities together, and automation. I'd like to see it work with
Forrestbot in order to keep sites fresh, I'd like to see it work with Maven
(to do their builds their way, and maybe their sites), and (as you know) I'd
like to see it work with Agora to pictorialize some interactions. I have
visions of extending Gump metadata to interact w/ other social network
metadata (e.g. FOAF) and hence automatically process non-code
inter-relationships. I know Nicola has some other ideas also. I feel if it
were opened up to such uses we'd see far far more.
I think Gump has two powerful things in it's favour for extending it (1) a
wealth of rich metadata (2) abilities to automatically work with
SCMs/external programs in an orderly fashion & report on results. That
combination is (IMHO) key.
> And I have the gut feeling that in order to have a meaningful FoG
> number, you need a history of both information. [but I have to think
> about that more because it seems to be a pretty complex
> graph-theory-related issue]
Yup, I recently added a to-do to status.xml to tie Gump to a historical
(likely RDBMS) database. Right now it has a simplistic DBM one, which works
fine (except on M$, ask the Python folks...) but I'd rather not see that
stretched further.
regards
Adam
Re: Nightly builds, Gump, Maven and repository issues.
Posted by Stefano Mazzocchi <st...@apache.org>.
On 30 Jan 2004, at 11:17, Adam R. B. Jack wrote:
> Mark wrote:
>
>> I wanted to to attempt to coordinate with Gump folks concerning
>> nightly
>> builds, the new maven repository on the apache server and other
>> interesting tidbits.
>
> I needed some time to mull this over before I responded. I see you
> have good
> intentions to try to pull a bunch of things together, and I respect
> that.
> That said, I don't think they should fit.
>
> Every time I try to match Gump with nightly builds, I come up against a
> philosophical problem which Stefan explained. Gump isn't doing nightly
> builds (against a stable base), it is doing nightly integration test
> builds
> (an unstable 'box of chocolates' :).
Yes. Many people believe that Gump is like Tinderbox but it's not. The
fact that it generates jars and javadocs is a *byproduct* of it.
At the same time, I think it would be useful to have both things: each
module should be built twice, the first time with "latest-greatest
dependencies" and the second time with "tested dependencies".
This might require some thinking but it would definately solve the
issues and yield two different signals: one for the box of chocolates,
one for the nightly built process.
I think they are both important.
And I have the gut feeling that in order to have a meaningful FoG
number, you need a history of both information. [but I have to think
about that more because it seems to be a pretty complex
graph-theory-related issue]
--
Stefano.
Re: Nightly builds, Gump, Maven and repository issues.
Posted by "Adam R. B. Jack" <aj...@trysybase.com>.
Mark wrote:
> I wanted to to attempt to coordinate with Gump folks concerning nightly
> builds, the new maven repository on the apache server and other
> interesting tidbits.
I needed some time to mull this over before I responded. I see you have good
intentions to try to pull a bunch of things together, and I respect that.
That said, I don't think they should fit.
Every time I try to match Gump with nightly builds, I come up against a
philosophical problem which Stefan explained. Gump isn't doing nightly
builds (against a stable base), it is doing nightly integration test builds
(an unstable 'box of chocolates' :). Gump's output jars aren't for general
use (few folks could/should live on the edge that much, each morning getting
a shifting base). IMHO they probably ought not have been distributed (in the
past) but for the future I see them as key to scaling a cascade of Gumps. As
such, although I think the Gump repository ought be open/available (no point
thinking for users) I envisage it's use only by other downstream Gumps.
regards,
Adam
Re: Nightly builds, Gump, Maven and repository issues.
Posted by Martin Cooper <ma...@apache.org>.
On Fri, 30 Jan 2004, Stefan Bodewig wrote:
> On Thu, 29 Jan 2004, Mark R. Diggory <md...@latte.harvard.edu>
> wrote:
>
> > 2.) There is the nightly build process on minotaur that Craig
> > McClanahan runs. This generates nightly builds under
> >
> > /www/cvs.apache.org/builds/...
>
> There used to be a nightly process on Sam's machine that placed
> Gump-built nightlies for various projects there as well.
>
> > 3.) There are the various continuous integration builds gump is
> > managing. These are producing jars and other distributables.
>
> Right now, all those Gump installations run on non-ASF hardware.
> There have been talks about setting up a dedicated machine for builds
> and Gump would use that, but there is not much yet.
>
> > It would be beneficial to see these various sources unified or
> > synced up in someway.
>
> Gump's goal probably is a bit different from what Craig's script
> does. I'm pretty sure that Craig will build against the latest
> releases of components the thing he wants to build depends on - while
> Gump will always build against the latest code.
I don't have a definitive list of what Craig's script builds, but I do
know that Commons components and Struts all build against the nightly
builds of each other, rather than against released builds.
--
Martin Cooper
>
> This means that Gump may fail to produce anything if any component you
> depend upon changes in a backwards incompatible way (and your
> compilation fails because of this). This is probably not acceptable
> for a nightly build systems unless you have very few or very
> stable/well-behaving components you depend upon.
>
> For things like Struts Gump couldn't replace Craig's script - and vice
> versa as Struts would be losing the early warning if things change.
>
> For projects that don't get built by Craig's script and are unlikely
> to be broken by components they depend on - say Ant - Gump could
> produce exactly the content that the repository needs.
>
> Gump can already publish the created artifacts and in fact it does so
> <http://gump.covalent.net/jars/>. It would be possibly to rsync the
> content from there (or any other machine running Gump) over to
> minotaur if we find a secure-enough way to do it. Some tweaking of
> the directory structure may be required for this to work as well.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: gump-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: gump-help@jakarta.apache.org
>
>
Re: Nightly builds, Gump, Maven and repository issues.
Posted by Stefan Bodewig <bo...@apache.org>.
On Thu, 29 Jan 2004, Mark R. Diggory <md...@latte.harvard.edu>
wrote:
> 2.) There is the nightly build process on minotaur that Craig
> McClanahan runs. This generates nightly builds under
>
> /www/cvs.apache.org/builds/...
There used to be a nightly process on Sam's machine that placed
Gump-built nightlies for various projects there as well.
> 3.) There are the various continuous integration builds gump is
> managing. These are producing jars and other distributables.
Right now, all those Gump installations run on non-ASF hardware.
There have been talks about setting up a dedicated machine for builds
and Gump would use that, but there is not much yet.
> It would be beneficial to see these various sources unified or
> synced up in someway.
Gump's goal probably is a bit different from what Craig's script
does. I'm pretty sure that Craig will build against the latest
releases of components the thing he wants to build depends on - while
Gump will always build against the latest code.
This means that Gump may fail to produce anything if any component you
depend upon changes in a backwards incompatible way (and your
compilation fails because of this). This is probably not acceptable
for a nightly build systems unless you have very few or very
stable/well-behaving components you depend upon.
For things like Struts Gump couldn't replace Craig's script - and vice
versa as Struts would be losing the early warning if things change.
For projects that don't get built by Craig's script and are unlikely
to be broken by components they depend on - say Ant - Gump could
produce exactly the content that the repository needs.
Gump can already publish the created artifacts and in fact it does so
<http://gump.covalent.net/jars/>. It would be possibly to rsync the
content from there (or any other machine running Gump) over to
minotaur if we find a secure-enough way to do it. Some tweaking of
the directory structure may be required for this to work as well.
Stefan