Ant+Ivy session at Apache Barcamp

[Antoine Levy Lambert <an...@gmx.de>]

Hi,

I have animated an improvised Ant+Ivy session at the Barcamp at 
ApacheCon US in Atlanta.

I got several interesting questions from participants :

- why do we not ship ant and ivy together ?

- which support is there in ant to take advantage of multicore machines 
to do work in parallel ?

- talk about ant and ivy best practices

Regards,

Antoine

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

AW: Ant+Ivy session at Apache Barcamp

[Jan Matèrne <ap...@materne.de>]

> I got several interesting questions from participants :

I don't have the official answer ;-)
But my few cents:


> - why do we not ship ant and ivy together ?

Never really thought of. Maybe because its different release times.
But what Ant could do is bundling the latest Ivy ... (or did we have some
discussions? There is something in the background of my mind ....)


> - which support is there in ant to take advantage of multicore machines
> to do work in parallel ?

I worked on a parallel executor in the sandbox some years ago - more because
I had an idea I want to check.
But I think, Ant would not profit from a multicore cpu, because IMO most
time of a build is I/O.


Jan 




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: AW: Ant+Ivy session at Apache Barcamp

[Antoine Levy-Lambert <an...@gmx.de>]

On 11/8/10 9:49 AM, Jan Matèrne wrote:
>> I would assume that if dependencies between targets are properly set,
>> and if it happens that the target dependency graph have two separates
>> branch which are merging at some point, Ant would then be able to run
>> those two separates branches in different threads, right ?
>>
>> I doubt there would be much improvement too with that kind of
>> parallelism, most of the target graphs I have seen are quite linear:
>> running test requires tests to be compiled, which requires the main
>> java to be compiled, which requires an ivy resolve.
>> When I saw real benefit of parallelism is when there are several
>> different projects to build, like within hudson, which is kind of out
>> of scope of ant itself (but maybe a fit in for EasyAnt :) ).
> The ParallelExecutor in the sandbox
> - sees each target as a building block which can be run in parallel
> - starts each target if all prior targets (depends-attribute) have finished
>
> Jan
>
Where I am working we are using a custom task to run JUnit and Selenium
tests in parallel.
This task is delegating to the <parallel/> task of ant, and either
creating one parallel subtask for each test suite,
or using an algorithm to group the test suites in sets taking
approximatively the same total time to run,
based upon stats of the previous run (from the file generated by
junitreport called TESTS-TestSuites.xml).

This way we are able to run 10 hours worth of JUnit test suites in one
hour, and 4 hours worth of selenium UTs in 40 minutes.

Our ant task is not implemented as an extension of the <junit/> ant
task, but fires a macro for each suite or group of suites, so it is only
loosely coupled with JUnit. Would such a design be desirable if this
task was contributed in ant ?

Concerning the builds, as Nicolas wrote, most of the improvement has to
come from the continuous integration server (CruiseControl where I
work). This is because we made the choice to create one CruiseControl
project for each RAD (Eclipse) project in our J2EE application.

If we had chosen to make one CruiseControl project running several ant
build files, then maybe we would have needed a parallel subant where the
build order is given by ivy inter-module dependencies, and also where
what is allowed to run in parallel with what would be also given by ivy.
Developing such a task would make sense. In fact Maven 3 has this built
in (building a multi module project with parallel threads).

Regards,

Antoine


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

AW: Ant+Ivy session at Apache Barcamp

[Jan Matèrne <ap...@materne.de>]

> I would assume that if dependencies between targets are properly set,
> and if it happens that the target dependency graph have two separates
> branch which are merging at some point, Ant would then be able to run
> those two separates branches in different threads, right ?
> 
> I doubt there would be much improvement too with that kind of
> parallelism, most of the target graphs I have seen are quite linear:
> running test requires tests to be compiled, which requires the main
> java to be compiled, which requires an ivy resolve.
> When I saw real benefit of parallelism is when there are several
> different projects to build, like within hudson, which is kind of out
> of scope of ant itself (but maybe a fit in for EasyAnt :) ).

The ParallelExecutor in the sandbox
- sees each target as a building block which can be run in parallel
- starts each target if all prior targets (depends-attribute) have finished

Jan


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Ant+Ivy session at Apache Barcamp

[Nicolas Lalevée <ni...@hibnet.org>]

Le 7 nov. 2010 à 07:28, Stefan Bodewig a écrit :

> On 2010-11-02, Antoine Levy Lambert wrote:
> 
>> - why do we not ship ant and ivy together ?
> 
> To some people our current distributions are too big already 8-)
> 
> There certainly is room for additional distribution formats like "the
> latest Ant bundled with the latest Ivy" that could be released whenever
> Ant or Ivy is released or "Ant plus the latest releases of selected
> Antlibs" or ...
> 
> Those things don't necessarily need to be part of our usual releases but
> could be put together separately.
> 
> In case of "latest releases" bundles, formally we wouldn't even need to
> vote on them.  In theory (and legaly) what we vote on is the source code
> we release and the binaries are only created for our users' convenience.

<advertising>
Maybe people looking for more prepackaged Ant are actually looking for something like EasyAnt ;)
</ad>

> 
>> - which support is there in ant to take advantage of multicore
>> machines to do work in parallel ?
> 
> From my experience with Gump I have to agree with Jan, building is I/O
> bound most of the time.  If you add parallelism here, it is likely going
> to slow things down rather than speed them up when two javac tasks
> compete for the disk.
> 
> Having said that there are things in your build that eat up time but are
> not I/O-bound.  Running tests for example.
> 
> In general Ant cannot know which parts of your build are good candidates
> to be spread accross cores and which it shouldn't run in parallel, I
> think an explicit <parallel> is better in this case.  Or the tasks would
> need to provide options, the <junit> task could provide an option to run
> tests in parallel for example.

I would assume that if dependencies between targets are properly set, and if it happens that the target dependency graph have two separates branch which are merging at some point, Ant would then be able to run those two separates branches in different threads, right ?

I doubt there would be much improvement too with that kind of parallelism, most of the target graphs I have seen are quite linear: running test requires tests to be compiled, which requires the main java to be compiled, which requires an ivy resolve.
When I saw real benefit of parallelism is when there are several different projects to build, like within hudson, which is kind of out of scope of ant itself (but maybe a fit in for EasyAnt :) ).

Nicolas


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org

Re: Ant+Ivy session at Apache Barcamp

[Stefan Bodewig <bo...@apache.org>]

On 2010-11-02, Antoine Levy Lambert wrote:

> - why do we not ship ant and ivy together ?

To some people our current distributions are too big already 8-)

There certainly is room for additional distribution formats like "the
latest Ant bundled with the latest Ivy" that could be released whenever
Ant or Ivy is released or "Ant plus the latest releases of selected
Antlibs" or ...

Those things don't necessarily need to be part of our usual releases but
could be put together separately.

In case of "latest releases" bundles, formally we wouldn't even need to
vote on them.  In theory (and legaly) what we vote on is the source code
we release and the binaries are only created for our users' convenience.

> - which support is there in ant to take advantage of multicore
> machines to do work in parallel ?

>From my experience with Gump I have to agree with Jan, building is I/O
bound most of the time.  If you add parallelism here, it is likely going
to slow things down rather than speed them up when two javac tasks
compete for the disk.

Having said that there are things in your build that eat up time but are
not I/O-bound.  Running tests for example.

In general Ant cannot know which parts of your build are good candidates
to be spread accross cores and which it shouldn't run in parallel, I
think an explicit <parallel> is better in this case.  Or the tasks would
need to provide options, the <junit> task could provide an option to run
tests in parallel for example.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org