You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by James Carpenter <na...@gmail.com> on 2005/04/11 02:04:22 UTC
Distributed multiproject builds
Questions are at the bottom.
==================================
Summary:
I am considering trying to build a rather large multiproject maven
build in a distributed fashion. I am looking for guidance on how to
best achieve this with a minimal amount of effort. In particular I am
looking for where to hook into the maven reactor during a multiproject
build.
==================================
Anticipated Design:
My anticipated approach is the following:
1) Share out the work directory on each developer's MS Windows based
workstation.
2) Build a simple JavaSpaces service that invokes an arbitrary maven
goal in a specified remote directory. The entity placed on the
JavaSpace by a developer's build will identify the developer's
machine, the developer's work directory, and the maven target to
invoke. The JavaSpace's service will either use it's own copy of
maven or it will leverage a copy of maven in the developer's work
directory in a known location. When the service completes running the
maven target it will report its completion (including success/failure
information) by placing an appropriate entity on the JavaSpace.
The JavaSpaces service would be installed on a handful of cheap PC's
dedicated to distributed maven builds.
3) Build an appropriate maven plugin that communicates with the
JavaSpaces service. This plugin should leverage maven's existing
multiproject functionality in order to understand the inter-project
dependencies. It is hope this can be done with minimal effort.
==================================
Notes:
1) I am only concerned with building projects in parallel. There is
no need to complicate the design by supporting distributed builds
within a project.
2) The developer's interaction with the distributed build should be
through maven. The usage should be similar to that of using the
multiproject plugin. The only difference is the multiproject-distrib
plugin would actually be farming work out to the build boxes.
3) It is expected the CPU and memory resources of the build servers
will be the greatest performance bottleneck and the disk I/O of the
developer's workstation will not initially be a problem. When/if disk
I/O becomes a problem the work directory can be kept in sync with a
few remote copies.
4) Current build time is dominated by JUnit tests not compilation.
There may be some value in compiling first and running tests second.
This will help parallelize the most time consuming portion of the
build. (Performance issues with JUnit tests are a nice problem to
have.)
5) The design chosen should be very simple to build and maintain. The
only reason I'm interested in solving this problem is the build cycle
has become painfully slow (15min or more) and is affecting agility. I
don't want to get sucked into build management, I just want the builds
to go faster. Complete builds under 1 or 2 minutes would make me very
happy. A continuous integration environment can make snapshots
available, but thats still not quite the same as running all the junit
tests across all the projects. (Although it would be better than 15
minute builds.)
==================================
Relevant knowledge:
1) I have lots of experience writing custom Ant tasks.
2) Very little development experience with maven plugins, but work
with people who have written several. (i.e. I don't really know Jelly
well.)
3) Don't know much about JavaSpaces, but the examples look pretty
simple. As my problem is fairly simple I can probably do what I need
without getting into the nitty gritty of JINI. It looks like
JavaSpaces is simple but learning JINI well is like learning EJBs
well. It does seem that a Linda engine is the easiest way to solve my
problem.
==================================
Questions:
1) Where do I hook into maven:reactor to figure out the build
dependencies? Where is the documentation for the maven:reactor goal?
Where is the file that tells me what java classes are associated with
the maven:reactor goal?.
2) Is there any open-source project (maven plugin) out there that has
already solved this problem? So far I havn't been able to find
anything close other than rant and AntSpaces
(http://www.oopsconsultancy.com/software/antspaces/).
3) If I can't find dedicated build boxes, I need to have the JavaSpace
service only run when a PC has it's screen locked. Any
recommendations here? I can't even think of how I can figure this out
inside of a Java application. Such low level system information isn't
typically exposed to Java. Maybe someone has already written
something that takes care of the problem for me. Could use guidance
here.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Distributed multiproject builds
Posted by James Carpenter <na...@gmail.com>.
Update:
Found maven:reactor source code at: org.apache.maven.jelly.tags.maven.ReactorTag
I still don't know how Jelly figures out this is a jelly tag but this
is obviously the appropriate code.
ViewCVS link to relevant directory is:
http://svn.apache.org/viewcvs.cgi/maven/maven-1/jelly-tags/trunk/src/main/java/org/apache/maven/jelly/tags/maven/?rev=112248
If you dig around you can see that ReactorTag delegates much of the work to:
org.apache.maven.jelly.tags.maven.WerkzDependencyResolver (same
directory as above)
This delegates the real work to com.werken.werkz.WerkzProject.
In an effort to find the source/javadoc for the werkz package I found:
http://werkz.werken.com/
and
http://werkz.sourceforge.net/
I can't seem to find the relevant content at http://werkz.werken.com
and what I find at http://werkz.sourceforge.net has slightly different
class names. It is apparent that the sourceforge content is related
but it doesn't seem to be exactly what maven is using.
Sadly from what I find at http://werkz.sourceforge.net there is no
ParallelSession implementation. I am hoping that the version maven is
actually using has already implemented such functionality. If not
then it is apparent I will have to write it if I want to get parallel
distributed builds working. Of course to do that I need to know
exactly what werken distribution maven is using.
I continue to need any guidance the reader can provide.
On Apr 10, 2005 7:04 PM, James Carpenter <na...@gmail.com> wrote:
> Questions are at the bottom.
>
> ==================================
> Summary:
>
> I am considering trying to build a rather large multiproject maven
> build in a distributed fashion. I am looking for guidance on how to
> best achieve this with a minimal amount of effort. In particular I am
> looking for where to hook into the maven reactor during a multiproject
> build.
>
> ==================================
> Anticipated Design:
>
> My anticipated approach is the following:
> 1) Share out the work directory on each developer's MS Windows based
> workstation.
>
> 2) Build a simple JavaSpaces service that invokes an arbitrary maven
> goal in a specified remote directory. The entity placed on the
> JavaSpace by a developer's build will identify the developer's
> machine, the developer's work directory, and the maven target to
> invoke. The JavaSpace's service will either use it's own copy of
> maven or it will leverage a copy of maven in the developer's work
> directory in a known location. When the service completes running the
> maven target it will report its completion (including success/failure
> information) by placing an appropriate entity on the JavaSpace.
>
> The JavaSpaces service would be installed on a handful of cheap PC's
> dedicated to distributed maven builds.
>
> 3) Build an appropriate maven plugin that communicates with the
> JavaSpaces service. This plugin should leverage maven's existing
> multiproject functionality in order to understand the inter-project
> dependencies. It is hope this can be done with minimal effort.
>
> ==================================
> Notes:
>
> 1) I am only concerned with building projects in parallel. There is
> no need to complicate the design by supporting distributed builds
> within a project.
>
> 2) The developer's interaction with the distributed build should be
> through maven. The usage should be similar to that of using the
> multiproject plugin. The only difference is the multiproject-distrib
> plugin would actually be farming work out to the build boxes.
>
> 3) It is expected the CPU and memory resources of the build servers
> will be the greatest performance bottleneck and the disk I/O of the
> developer's workstation will not initially be a problem. When/if disk
> I/O becomes a problem the work directory can be kept in sync with a
> few remote copies.
>
> 4) Current build time is dominated by JUnit tests not compilation.
> There may be some value in compiling first and running tests second.
> This will help parallelize the most time consuming portion of the
> build. (Performance issues with JUnit tests are a nice problem to
> have.)
>
> 5) The design chosen should be very simple to build and maintain. The
> only reason I'm interested in solving this problem is the build cycle
> has become painfully slow (15min or more) and is affecting agility. I
> don't want to get sucked into build management, I just want the builds
> to go faster. Complete builds under 1 or 2 minutes would make me very
> happy. A continuous integration environment can make snapshots
> available, but thats still not quite the same as running all the junit
> tests across all the projects. (Although it would be better than 15
> minute builds.)
>
> ==================================
> Relevant knowledge:
>
> 1) I have lots of experience writing custom Ant tasks.
>
> 2) Very little development experience with maven plugins, but work
> with people who have written several. (i.e. I don't really know Jelly
> well.)
>
> 3) Don't know much about JavaSpaces, but the examples look pretty
> simple. As my problem is fairly simple I can probably do what I need
> without getting into the nitty gritty of JINI. It looks like
> JavaSpaces is simple but learning JINI well is like learning EJBs
> well. It does seem that a Linda engine is the easiest way to solve my
> problem.
>
> ==================================
> Questions:
>
> 1) Where do I hook into maven:reactor to figure out the build
> dependencies? Where is the documentation for the maven:reactor goal?
> Where is the file that tells me what java classes are associated with
> the maven:reactor goal?.
>
> 2) Is there any open-source project (maven plugin) out there that has
> already solved this problem? So far I havn't been able to find
> anything close other than rant and AntSpaces
> (http://www.oopsconsultancy.com/software/antspaces/).
>
> 3) If I can't find dedicated build boxes, I need to have the JavaSpace
> service only run when a PC has it's screen locked. Any
> recommendations here? I can't even think of how I can figure this out
> inside of a Java application. Such low level system information isn't
> typically exposed to Java. Maybe someone has already written
> something that takes care of the problem for me. Could use guidance
> here.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: Distributed multiproject builds
Posted by Vincent Massol <vm...@pivolis.com>.
Cool. This looks very similar to what I have described here:
http://tinyurl.com/3l8eh
Some comments/ideas:
1/ Personally if I were to develop such a system I would make it relatively
independent of the underlying build tool (Maven, Ant or anything else). It
would sit on top of any of those tools. The only part where it would need
integration is:
- calling the build
- getting build results
2/ I'd look at integrating an existing project working on this topic. I can
think of 3:
- CruiseControl. They have started on this and someone has already
contributed code
- DamageControl
- Continuum
3/ In Maven land, I wouldn't build it on top of the reactor. I'd rather look
at continuum.
Thanks
-Vincent
> -----Original Message-----
> From: James Carpenter [mailto:nawkboy@gmail.com]
> Sent: lundi 11 avril 2005 02:04
> To: dev@maven.apache.org
> Subject: Distributed multiproject builds
>
> Questions are at the bottom.
>
> ==================================
> Summary:
>
> I am considering trying to build a rather large multiproject maven
> build in a distributed fashion. I am looking for guidance on how to
> best achieve this with a minimal amount of effort. In particular I am
> looking for where to hook into the maven reactor during a multiproject
> build.
>
> ==================================
> Anticipated Design:
>
> My anticipated approach is the following:
> 1) Share out the work directory on each developer's MS Windows based
> workstation.
>
> 2) Build a simple JavaSpaces service that invokes an arbitrary maven
> goal in a specified remote directory. The entity placed on the
> JavaSpace by a developer's build will identify the developer's
> machine, the developer's work directory, and the maven target to
> invoke. The JavaSpace's service will either use it's own copy of
> maven or it will leverage a copy of maven in the developer's work
> directory in a known location. When the service completes running the
> maven target it will report its completion (including success/failure
> information) by placing an appropriate entity on the JavaSpace.
>
> The JavaSpaces service would be installed on a handful of cheap PC's
> dedicated to distributed maven builds.
>
> 3) Build an appropriate maven plugin that communicates with the
> JavaSpaces service. This plugin should leverage maven's existing
> multiproject functionality in order to understand the inter-project
> dependencies. It is hope this can be done with minimal effort.
>
> ==================================
> Notes:
>
> 1) I am only concerned with building projects in parallel. There is
> no need to complicate the design by supporting distributed builds
> within a project.
>
> 2) The developer's interaction with the distributed build should be
> through maven. The usage should be similar to that of using the
> multiproject plugin. The only difference is the multiproject-distrib
> plugin would actually be farming work out to the build boxes.
>
> 3) It is expected the CPU and memory resources of the build servers
> will be the greatest performance bottleneck and the disk I/O of the
> developer's workstation will not initially be a problem. When/if disk
> I/O becomes a problem the work directory can be kept in sync with a
> few remote copies.
>
> 4) Current build time is dominated by JUnit tests not compilation.
> There may be some value in compiling first and running tests second.
> This will help parallelize the most time consuming portion of the
> build. (Performance issues with JUnit tests are a nice problem to
> have.)
>
> 5) The design chosen should be very simple to build and maintain. The
> only reason I'm interested in solving this problem is the build cycle
> has become painfully slow (15min or more) and is affecting agility. I
> don't want to get sucked into build management, I just want the builds
> to go faster. Complete builds under 1 or 2 minutes would make me very
> happy. A continuous integration environment can make snapshots
> available, but thats still not quite the same as running all the junit
> tests across all the projects. (Although it would be better than 15
> minute builds.)
>
> ==================================
> Relevant knowledge:
>
> 1) I have lots of experience writing custom Ant tasks.
>
> 2) Very little development experience with maven plugins, but work
> with people who have written several. (i.e. I don't really know Jelly
> well.)
>
> 3) Don't know much about JavaSpaces, but the examples look pretty
> simple. As my problem is fairly simple I can probably do what I need
> without getting into the nitty gritty of JINI. It looks like
> JavaSpaces is simple but learning JINI well is like learning EJBs
> well. It does seem that a Linda engine is the easiest way to solve my
> problem.
>
> ==================================
> Questions:
>
> 1) Where do I hook into maven:reactor to figure out the build
> dependencies? Where is the documentation for the maven:reactor goal?
> Where is the file that tells me what java classes are associated with
> the maven:reactor goal?.
>
> 2) Is there any open-source project (maven plugin) out there that has
> already solved this problem? So far I havn't been able to find
> anything close other than rant and AntSpaces
> (http://www.oopsconsultancy.com/software/antspaces/).
>
> 3) If I can't find dedicated build boxes, I need to have the JavaSpace
> service only run when a PC has it's screen locked. Any
> recommendations here? I can't even think of how I can figure this out
> inside of a Java application. Such low level system information isn't
> typically exposed to Java. Maybe someone has already written
> something that takes care of the problem for me. Could use guidance
> here.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
_________________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en fran�ais !
Yahoo! Mail : http://fr.mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org