You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by Alex Iliev <al...@gmail.com> on 2009/01/27 22:51:44 UTC
buildlist/subant in parallel
I was thinking that the combination of ivy:buildlist and subant is a
perfect candidate for parallelization: the dependency graph tells us
exactly which Ivy modules can be built in parallel. I'd be interested
in implementing some such functionality. Any thoughts from the list on
whether something like this exists or is in progress?
Cheers
alex
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: buildlist/subant in parallel
Posted by Alex Iliev <al...@gmail.com>.
> I have written an executor which tries to parallelize a "normal" Ant
> build.
> http://svn.apache.org/repos/asf/ant/sandbox/parallelexecutor/
Thanks! I'll check it out.
What is the policy for using Java 5 language features and APIs in ant?
I wouldn't want to use wait/notify here - would have to be at least
the backport of util.concurrent.
alex
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
AW: buildlist/subant in parallel
Posted by Ja...@rzf.fin-nrw.de.
>I'm not aware of any implementation, but I think it might be an
>excellent idea. It would be a good step in the direction of
>concurrent build.
>
>However, note that implement the ivy-parallel-subant is not the only
>thing you will have to do.
>You might also have the ensure that your sub project can indeed be
>build in parallel. The kind of issue you might have to solve are :
>- Thread safety issue in some tasks
>- Current access to some external ressources (database, repository,
>file system, ...)
>
>Anyway, I'm sure that concurrent build is the direction we have to
>take. And an ivy-parallel-subant would be a good first step.
>
>Gilles Scokart
>
>
>
>2009/1/27 Alex Iliev <al...@gmail.com>:
>> I was thinking that the combination of ivy:buildlist and subant is a
>> perfect candidate for parallelization: the dependency graph tells us
>> exactly which Ivy modules can be built in parallel. I'd be interested
>> in implementing some such functionality. Any thoughts from
>the list on
>> whether something like this exists or is in progress?
I have written an executor which tries to parallelize a "normal" Ant
build.
http://svn.apache.org/repos/asf/ant/sandbox/parallelexecutor/
Maybe you can get some ideas ;-)
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: buildlist/subant in parallel
Posted by Alex Iliev <al...@gmail.com>.
> You might also have the ensure that your sub project can indeed be
> build in parallel. The kind of issue you might have to solve are :
> - Thread safety issue in some tasks
> - Current access to some external ressources (database, repository,
> file system, ...)
I read about the non-reentrancy of some major tasks, and one can't
assume anything about third-party tasks, so I was thinking that
running each sub-build in a separate JVM is the way to go, or at least
to start. JVM-pools can be an optimization later on.
alex
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: buildlist/subant in parallel
Posted by Gilles Scokart <gs...@gmail.com>.
I'm not aware of any implementation, but I think it might be an
excellent idea. It would be a good step in the direction of
concurrent build.
However, note that implement the ivy-parallel-subant is not the only
thing you will have to do.
You might also have the ensure that your sub project can indeed be
build in parallel. The kind of issue you might have to solve are :
- Thread safety issue in some tasks
- Current access to some external ressources (database, repository,
file system, ...)
Anyway, I'm sure that concurrent build is the direction we have to
take. And an ivy-parallel-subant would be a good first step.
Gilles Scokart
2009/1/27 Alex Iliev <al...@gmail.com>:
> I was thinking that the combination of ivy:buildlist and subant is a
> perfect candidate for parallelization: the dependency graph tells us
> exactly which Ivy modules can be built in parallel. I'd be interested
> in implementing some such functionality. Any thoughts from the list on
> whether something like this exists or is in progress?
>
> Cheers
>
> alex
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
AW: buildlist/subant in parallel
Posted by Ja...@rzf.fin-nrw.de.
>> > Of particular concern is logging: your logger must be able to deal
>with the
>> > fact that you have projects running in parallel or your
>logs will be
>an
>> > incomprehensible jumble.
>>
>> Thanks for the tip! Prefixing logging lines, and/or writing to
>> separate log files per project look like reasonable options here. Was
>> your solution something like this? (if you can say :)
>
>I can't (yet) talk about the details of our solution; sorry.
Regarding to logging I thought about prefexing with the target name, but
havent implemented it in the ParallelExecutor.
Regarding the Java version - it depends if it should be in th Ant core
(JDK 1.3 ATM) or external (like the sandbox). In the later case you are
free to use any version. Just put a note in the readme.
Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: buildlist/subant in parallel
Posted by Jeffrey E Care <ca...@us.ibm.com>.
> > Of particular concern is logging: your logger must be able to deal
with the
> > fact that you have projects running in parallel or your logs will be
an
> > incomprehensible jumble.
>
> Thanks for the tip! Prefixing logging lines, and/or writing to
> separate log files per project look like reasonable options here. Was
> your solution something like this? (if you can say :)
I can't (yet) talk about the details of our solution; sorry.
Re: buildlist/subant in parallel
Posted by Alex Iliev <al...@gmail.com>.
On Wed, Jan 28, 2009 at 2:50 AM, Jeffrey E Care <ca...@us.ibm.com> wrote:
> Of particular concern is logging: your logger must be able to deal with the
> fact that you have projects running in parallel or your logs will be an
> incomprehensible jumble.
Thanks for the tip! Prefixing logging lines, and/or writing to
separate log files per project look like reasonable options here. Was
your solution something like this? (if you can say :)
alex
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: buildlist/subant in parallel
Posted by Jeffrey E Care <ca...@us.ibm.com>.
We don't use Ivy, but we do have our own set of extensions that has some
similar concepts in terms of multiple projects with dependencies between
them.
We use multithreaded build dispatching based on a dependency graph & let
me tell you: it's not an easy thing to get right. I've had to make several
changes to core Ant classes to get this stuff working properly.
Of particular concern is logging: your logger must be able to deal with
the fact that you have projects running in parallel or your logs will be
an incomprehensible jumble.
Not trying to discourage anyone from giving this a go, but it's certainly
not for the faint hearted.
____________________________________________________________________________________________
Jeffrey E. (Jeff) Care
carej@us.ibm.com
IBM WebSphere Application Server
WAS Release Engineering
Alex Iliev <al...@gmail.com> wrote on 01/27/2009 04:51:44 PM:
> [image removed]
>
> buildlist/subant in parallel
>
> Alex Iliev
>
> to:
>
> dev
>
> 01/27/2009 04:53 PM
>
> Please respond to "Ant Developers List"
>
> I was thinking that the combination of ivy:buildlist and subant is a
> perfect candidate for parallelization: the dependency graph tells us
> exactly which Ivy modules can be built in parallel. I'd be interested
> in implementing some such functionality. Any thoughts from the list on
> whether something like this exists or is in progress?
>
> Cheers
>
> alex
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
Re: buildlist/subant in parallel
Posted by Petar Tahchiev <pa...@gmail.com>.
Hi Alex,
I am not aware of anything in progress right now, but I think the idea is
worth a shot.
+1 from me :-)
Cheers, Petar
2009/1/27 Alex Iliev <al...@gmail.com>
> I was thinking that the combination of ivy:buildlist and subant is a
> perfect candidate for parallelization: the dependency graph tells us
> exactly which Ivy modules can be built in parallel. I'd be interested
> in implementing some such functionality. Any thoughts from the list on
> whether something like this exists or is in progress?
>
> Cheers
>
> alex
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
>
>
--
Regards, Petar!
Karlovo, Bulgaria.
- - - - - - - -
| Author @ Manning Publications.
| Senior Solution Architect @ Unic
| BGJUG-Bulgarian Java User Group Leader.
| Apache Maven Developer.
| Apache Jakarta PMC member.
| Jakarta Cactus Lead Developer.
| Blogger: http://weblogs.java.net/blog/paranoiabla/
- - - - - - - -
Public PGP Key at:
https://keyserver1.pgp.com/vkd/DownloadKey.event?keyid=0x19658550C3110611
Key Fingerprint: A369 A7EE 61BC 93A3 CDFF 55A5 1965 8550 C311 0611
Re: buildlist/subant in parallel
Posted by "Ivan \"Rambius\" Ivanov" <ra...@yahoo.com>.
Hello Alex,
It is possible to introduce parallelism with ivy:buildlist and subant. I have not done it in this way explicitly. Instead we are using a customized version of CruiseControl that is able to sort out the projects using their ivy descriptors and then we run the CruiseControl builds in a couple of threads so some of them are built in parallel. It is not bullet proof method, however.
Regards
Rambius
Tangra Mega Rock: http://www.radiotangra.com/
--- On Tue, 1/27/09, Alex Iliev <al...@gmail.com> wrote:
> From: Alex Iliev <al...@gmail.com>
> Subject: buildlist/subant in parallel
> To: dev@ant.apache.org
> Date: Tuesday, January 27, 2009, 11:51 PM
> I was thinking that the combination of ivy:buildlist and
> subant is a
> perfect candidate for parallelization: the dependency graph
> tells us
> exactly which Ivy modules can be built in parallel. I'd
> be interested
> in implementing some such functionality. Any thoughts from
> the list on
> whether something like this exists or is in progress?
>
> Cheers
>
> alex
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
> For additional commands, e-mail: dev-help@ant.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org